RSoXS accessing data

<<Back to RSoXS

Accessing Data

From Tiled

Example of path to personal JupyterHub directory

cd /nsls2/users/pketkar

From channel archiver

Use this to access continuously archived data that was not collected during a scan or associated with a scan ID.

1. On the RSoXS control computer, start phoebus. On the top left, go to Applications --> Display --> Data Browser.
2. Viewing data: In the Data Browser, right click and select Add PV (PV = process variable). Enter the name of the PV to display. The time axis can be adjusted by clicking on the clock icon on the top panel (open start/end time dialog box).
3. Exporting data: In the Data Browser with the displayed data, right click and select Open Data Export Panel.
   • Samples to Export: Raw Archived Data
   • Format: Spreadsheet, Tabular, Default format (for numbers on the order of 1e0 or larger) or Exponential notation with 6 digits (for numbers significantly smaller than 1e0)
   • Export into selected location (e.g., /nsls2/data/sst/legacy/RSoXS/ArchivedData).

Suitcased data and raw images

Using Globus (globus.nsls2.bnl.gov)

1. Download a Globus personal client
   1. Go to https://www.globus.org/globus-connect-personal
   2. Install personal client for Windows or Mac
   3. Set a Collection Name (e.g., NSLS2Data)
2. Go to https://app.globus.org/
3. Do first-time setup
   1. Select Brookhaven National Laboratory so that you can log in using your BNL credentials
   2. Complete sign up during the first time you log in.
   3. Click "Allow" to allow Globus to manage files.
4. In File Manager, select Collection as NSLS2, and enter the path for the beam time data.
5. Select "Transfer or Sync to..." and enter your collection name for your personal client and the path on your local computer where you want to save the data.
6. Select the desired files and click "Start" to transfer them to your local computer.

Using ssh

Access to data is straightforward in the new system, and if you know what you are doing, you need to sftp into sftp.nsls2.bnl.gov by ssh tunneling through ssh.nsls2.bnl.gov, logging into each with your BNL logon and DUO confirmation, and browsing to “/nsls2/data/sst/proposals/”. Navigate to the correct cycle and proposal numbers. Suitcased data should be saved in this directory, and raw images would be saved in subdirectories with the same name as the detector; there may be subfolders for the year, month, or day.  I will give basic directions for Mac and Windows below.  I assume if you have linux, you can already figure this part out.

See change to SSH Gateways on November 14, 2024: https://www.bnl.gov/announcement.php?q=11166

 

Mac

1. Download and install Filezilla, which is the easiest sftp client I have found got mac OS. https://filezilla-project.org/download.php?platform=osx
2. Open a Terminal (apple+ space and start typing in terminal and enter)
3. In the terminal type: “ssh -D 1234 -q -C -N egann@ssh.nsls2.bnl.gov” replacing “egann” with your BNL username. Enter your password and complete the DUO authentication. The terminal should just stay blank. If it returns to a prompt the connection has closed, and you need to repeat the above command. The connection will likely timeout after a few minutes.
4. In filezilla, open FileZilla -> Settings -> Generic proxy.
5. Choose SOCKS 5 with

• the proxy host: localhost,
• the proxy port: 1234
• and your BNL username and password
• click “ok”
• In Filezilla create a new Site in the “file-> Site Manager”

• protocol: sftp
• host: sftp.nsls2.bnl.gov
• enter your BNL username and password
• in “Advanced” tab on the site manager, enter the default local directory as:

• “/nsls2/data/sst/proposals/”
• Click “Connect”
• Browse to your data directory

A more detailed guide to FireZilla is available here: https://nsls2.github.io/bmm-beamline-manual/data.html#a-guide-with-pictures

Windows

1. Download and install WinSCP, which is the easiest way to do this in windows. https://winscp.net/eng/index.php
2. create a new session to Host name: sftp.nsls2.bnl.gov, with your BNL username and password
3. click the advanced button (or edit and then advanced)
4. Choose the Connection -> Tunnel selection on the left and Check the “Connect through SSH Tunnel” checkbox, and enter in ssh.nsls2.bnl.gov as the hostname, and your BNL account information once again.
5. 
6. 
7. You might get a message that looks like the one below. It is okay to type yes to allow your machine to write the new fingerprint. For more information, see: https://docs.nsls2.bnl.gov/docs/remote/ssh.html feynman@host ~> ssh feynman@ssh.nsls2.bnl.gov The authenticity of host 'ssh.nsls2.bnl.gov (192.203.218.31)' can't be established. ED25519 key fingerprint is SHA256:41IwO0No8QHm6LXq7lVo74Gbk2dJZskbLWIPtD28Ay0. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])?
8. Choose the “Directories” option on the left menu, and enter “/nsls2/data/sst/proposals/” in the remote directory
9. 

Click OK, then Save and Login on the previous window.

All platforms

Download the entire directory structure of your experiment to your local computer, generally this is the directory with your proposal ID and institution you will find on the remote server. Please refrain from accessing other users data unless you have permission.

NOTE: If you aren’t seeing any data directories, but seem to be in a folder that is named your BNL login, then you need to browse to “/nsls2/data/sst/proposals” either you have forgotten to set that as your default directory, or you have overridden it somehow. Manually browse to this directory to find your data!

In WinSCP for instance:

and type in

Monitoring suitcasing on Prefect

Prefect is used to run post-processing workflows on data generated at the beamline. Currently, it is used to suitcase raw data into CSV or TIFF files and save these files into the proposal directory, but further processing (e.g., double normalization) is possible.

To monitor the workflows, the beamline scientist can use single sign-on to log into https://app.prefect.cloud/ (https://docs.nsls2.bnl.gov/docs/how-to/adaptive.html#configure-prefect) using BNL credentials. Navigate to the prod workspace. Then, navigate to "Flow Runs" on the lefthand panel. The runs can be filtered by the rsoxs-work-pool-docker work pool and also by the date range. Completed runs should show up as green dots. Occasionally, some or many runs may fail and show up as orange or red dots. To start troubleshooting, compare a completed run to a failed run to narrow down what step might have failed. A small number of failed runs could be rerun by clicking the Retry button or Copy to new run. If a larger number of runs failed, contact the beamline scientist to rerun the failed scan IDs through a custom Jupyter notebook. Abigail (Abby) Giles can provide further assistance if needed.