Beamline shutdown/startup during a synchrotron shutdown or power shutdown
Synchrotron shutdowns occur at the end of each cycle (3 times per year). A power shutdown typically occurs every 1-2 years.
Beamline shutdown
Turn off oxygen leak on the mirrors and gratings. This includes M1, L1, L2A/L2B, PGM, M3AB/M3C, and M4C/M4D. For each mirror/grating assembly, close the valve such that it is finger-tight. Check the assembly gauges to ensure that the pressure has dropped. Note down the pressure in the oxygen line gauge (oxygen pressure in the line outside the assemblies) to ensure that it does not drop over the shutdown; if there is a drop, fix the leak in the tank or tubing.Example M1 mirror assembly. Red box: valve for oxygen leak into mirror/grating assembly. Yellow box: pressure gauge for oxygen line prior to entering the mirror/grating assembly.
Turn off cameras (e.g., switch off WAXS CCD camera). Also, slowly turn down high-voltage hardware (e.g., PEY NEXAFS chamber components), switch off hardware, and unplug from the beamline. Even when a power shutdown is not scheduled, power dips can cause issues with these equipment.
Ensure hardware is withdrawn (e.g., RSoXS post-beam time motor positions after running nmode) . Close manual load lock valves (e.g., RSoXS solid samples, TEM).
Isolate measurement stations. For the RSoXS chamber, close PSH10, GV27A, and GV28.
Ion pumps, turbo pumps, and roughing pumps are left on. Ensure that they are set to autostart after power is available.Yellow box: Rocker switch on backing pump for solid sample load lock set to auto-restart
After checking with all other beamline scientists, stop IOCs (1, 2, 3, and 7 for RSoXS). Switch off their respective motor controllers. It is important to check with the other beamline scientists so that prematurely switching off an IOC does not disrupt their running hardware. This step is essential for power shutdowns and optional, but recommended, for synchrotron shutdowns. If there were an unexpected loss of power during a synchrotron shutdown, it is safer to have this hardware turned off beforehand.
After all hardware is in its shutdown state, shut off workstation computers (e.g., RSoXS control and RSoXS Analysis computers). This step is recommended for a power shutdown, but not essential for a beamline shutdown.
Beamline startup
Check all ion pump gauges and ensure that there is sufficient vacuum in the beamline. Pressures should be on the order of 1e-10 to 1e-8. Pressures of 1e-7 indicates a leak. Provided that a power shutdown/loss was brief and the beamline is properly sealed, the beamline should pump down to the necessary pressure. However, if it is difficult to pump down, a turbo pump can be connected near the location that requires additional pumping (e.g., the Izero/DM [diagnostic module] region directly upstream of the RSoXS chamber). Sections with turbopumps (e.g. the main RSoXS chamber) may need the roughing to run a while before the turbos should be restarted.
If needed, ion gauges can be started manually from the gauge racks by navigating to the desired channel and turning on the sensor. Use the vacuum diagram in phoebus to ensure that all necessary gauges are turned on.The green light indicates which channel is activated. The up/down buttons can be used to navigate to different channels. The Sensor On/Off button can be used to turn on/off an ion gauge. The names of the streams are labeled on the left. The last number on the stream (e.g., 136 in "VS14-PGM-136") indicates the wire number connecting the gauge to the beamline hardware. In case the rest of the label is unclear, the wire number can be traced back to the beamline hardware.
Open/close gate valves via phoebus (EPICS) whenever possible. If needed, the valves also can be manually open, but this might leave them open during a future power outage when they should otherwise automatically close.
Ensure the pressure in oxygen leak lines has not dropped since the beginning of the shutdown. Open the oxygen leak valves to the mirror/grating assemblies detailed in the beamline shutdown procedure.
In general, turn on all physical equipment first before starting back up the IOCs. This includes the motor controllers, detector cameras, etc.
Most process variable names can be found in phoebus by going to the "SST Launch" main page. The process variable name can be found by hovering over its respective icon. Right click and select "Copy PV to Clipboard" to use this PV in Bluesky scripts.
Undulator parameters can are located in the "Insertion device" menu on the "SST Launch" main page. Scroll to the right and select "7 SST". The "C07 EPU60" parameters correspond to the soft branch, which is used for RSoXS.
Temperatures and corresponding water flow rates for beamline components (e.g., mirrors) are located in the "EPS" --> "M" menu.
Beam characterization and alignment
This procedure can be used to align the beam from the EPU to the RSoXS station. It is recommended to perform this procedure at least the beginning of each cycle and after any significant ring current changes to ensure that the central beam from the EPU is selected and it is robust to upstream beam motions. Unfortunately, the jogging of mirror positions is not repeatable, so going back to an absolute parameter value will not necessarily give the optimal flux.
First, note the original EPU front-end slit settings and mirror parameters (e.g., mir1.pitch.read(), mir3.pitch.read(), mir3.x.read() in Bluesky) prior to adjustment. If needed, these motors also can be found in Phoebus by going to File --> Open --> /opt/css/opi/production/cs-studio-xf/07id/edl_conv. M1 motors can be accessed in /MC02/6Pod_TopLevel_M1.bob, and M3 motors can be accessed /MC05/6Pod_TopLevel_M3ABC.bob.
View the beam on the FS1 white beam stop (and BPM1 camera, immediately downstream of M1) by moving M1 lateral motion in the negative direction (RE(bps.mv(mir1.x, -5))).
Open the EPU60 front-end slit vertical and horizontal apertures to view the full beam. Use FS1 to assess the full beam size such that opening the front-end slit apertures does not increase the size of the beam image. Images can be captured using RE(bp.count([FS1_cam], num=1, delay=0)).
Align and characterize the beam using the FS6 screen (and BPM6), which is immediately upstream of photon shutter 4 prior to exiting the hutch. Characterizing the beam at this location can reveal the effects of M1 on the beam.
Move the FS6 vertical motor to -17 such that the screen would be in the beam path. This motor control GUI can be found in phoebus by going to File --> Open --> /opt/css/opi/production/cs-studio-xf/07id/edl_conv/MC01/FS06_MainPanel.bob
Adjust M1 parameters (e.g., RE(bps.mv(mir1.x, -0.8)), RE(bps.mv(mir1.pitch, 0.5))) such that the beam becomes visible on FS6, the beam size is maximized, and the beam intensity is homogeneous across the beam area. Different material stripes (gold, carbon, and nickel) can be accessed by moving mir1.y. If the oxygen bleed is on, DO NOT move to the carbon strip or expose this strip to the beam while moving to another strip; otherwise, the oxygen bleed can degrade the carbon coating. If it is necessary to change the strip, fully retract the mirror (mir1.x = -5), fully close the EPU60 front-end slits (verifying on FS1), move to the necessary mir1.y, and then set mir1.x and EPU60 front-end slit parameters to the desired settings in that order. If moving to the carbon strip, turn off oxygen bleed before making any changes to mir1.y.
Adjust the EPU60 front-end slit aperture sizes and centers such that the aperture centers on the centers of the beam area. Open the front-end slit apertures for further downstream characterization. Ideally, the front-end slit apertures should be adjusted using FS1, but it is challenging to view the full beam there.
Adjust the FOE slits which are past FS1 to remove lower-intensity beam reflections that may possibly come from a bending magnet. This can be accessed in Phoebus by going to File --> Open --> /opt/css/opi/production/cs-studio-xf/07id/edl_conv/MC03/Slt01_MainPanel.bob
Return the FS6 motor position to -1.98, where it is set during normal operation.
Align and characterize the beam using the FS7 pink beamstop (and BPM7 camera), which is immediately downstream of M3. Comparing the beam image from FS7 with that from FS6 can reveal the effects of M2 and the PGM on the beam.
Retract M3 (RE(bps.mv(mir3.x, 0))).
First view the "zero-order" beam by setting the M2 and PGM grating angles equal to each other (e.g., -8 to reduce overall flux). If you want to move both the grating and the mirror in the negative direction, move the mirror first; if you want to move the grating and mirror in the positive direction, move the grating first. The current motor angles can be read using the mirror2.read() and grating.read() commands in Bluesky. Similarly, the x positions of these components can be read using mirror2x.read() and gratingx.read().
Then view the monochromatic beam by selecting an energy (e.g., e 270).
For both the pink and monochromatic beams, M1 parameters can be adjusted if needed.
Align and characterize the beam using the yag screen (and Scr1 camera) on the RSoXS diagnostic module, which is immediately downstream of RSoXS slit 1. Due to the angle of the yag, it may appear as if there are two spots of light on the yag, but the second spot is an artifact due to refraction. At the moment, the RSoXS sample viewer camera and the diagnostic module camera use the same channel. When the RSoXS sample camera is in use, plug the Cam-S > rg-c1_pp1:46 ethernet cable into the "RSoXS Cam-S" slot 46 in the hardware cabinet on the SST-2 side of the beamline. When the diagnostic module camera is in use, plug this cable into the "RSoXS Cam-I" slot 45.
Move M3 (RE(bps.mv(mir3.x, 24))) and the RSoXS yag (RE(Izero_screen()) or RE(bps.mv(izero_y, 2))) into the beam path.
With RSoXS slit 1 apertures fully open (e.g., RE(bps.mv(slits1.vsize, 10, slits1.hsize, 10))), adjust M3 parameters to center the beam on the bullseye pattern of the yag.
Close the EPU60 front-end slits to their desired aperture sizes and ensure that the apertures are still centered on the beam. It may be necessary to go off the vertical center to get the beam center closer to the center of the RSoXS WAXS camera. So far, adjusting mirror y, roll, and yaw has not successfully moved the beam vertically.
Close the RSoXS slit 1 apertures to the sizes used during normal operation to ensure that the beam is centered on the apertures. It can be helpful to use thin strips of tape or a sticky note to mark the center of the slit 1 apertures on the screen to ensure that the brightest part of the beam is positioned there.
[Not tested yet] Characterize the beam size using slit drain currents.
Load a NEXAFS configuration (e.g., WAXSNEXAFS), which will move RSoXS slits 1 to their positions during normal operation and bring the I0 gold mesh into the beam path. Make minor adjustments to the M3 pitch (~0.02 increments) and possibly M3 X (~0.05 increments) to maximize the I0 signal. It is usually good to keep RSoXS slits 1 positions constant such that the same portion of the I0 mesh is hit by X-rays and grows a steady layer of carbon contamination. If the X-rays keep hitting different regions of the I0 mesh, the actual beam intensity will be convoluted with the apparent signal due to the transient significant growth of carbon. After maximizing the signal, it may be necessary to adjust data collection parameters in the electrometers to reduce noise in the signal and ensure that saturation will not occur.
Optimize RSoXS slits such that the apertures are centered on the beam and blocking unwanted scattering.
Very important: attenuate the beam intensity by using ALL of the following settings. Do not run open beam scans with flux larger than what it would be at these settings.
Load the configuration (RE(load_configuration("WAXS_OpenBeamImages"))), in which the energy is reduced (e 150), aperture size of Precision Slit C (Slt:11 - YGap in PyDM) is reduced from -3.05 (default) to -0.01 (RE(bps.mv(Exit_Slit, -0.01))), and RSoXS diagnostic module (and I0 mesh) is out of the beam path.
Reduce the exposure time (exp 0.01)
It might be good to further set the PGM at 1200 l/mm, although CFF might change (TODO: check on this).
Take a snapshot image, and ensure that no beam can be seen while the WAXS beamstop is covering the direct beam.
Then, withdraw the WAXS beamstop in PyDM to a position of 3 (RE(bps.mv(BeamStopW, 3))), reduce binning (binning 1), and take another snapshot. An open beam image should be visible now
Start reducing the slit 2 aperture sizes and adjust the center so that scattering lobes are decreased.
Move the beamstop back into the beam path (RE(bps.mv(BeamStopW, 69.9))) and increase binning to default (binning 4), and take another snapshot.
Adjust the beamstop position using the manual port aligners such that the beamstop is at the center of the beam.
Gradually open Precision Slit C and narrow the slit 3 apertures such that the scattering from Slits 3 is visible on the camera. Adjust Slit 3 aperture sizes and centers.
[Move to a NEXAFS configuration and make minor adjustments to the M3 pitch (~0.02 increments) and possibly M3 X (~0.05 increments) to maximize the WAXS beamstop flux. Also observe the Izero in phoebus to ensure that flux is not being lost. The optimal beamstop diode should be ~5e-8 at 270 eV. The optimal flux for Izero might not be the same as the beamstop, but optimize for the beamstop, and ensure Izero still has sufficient flux. It may be necessary to adjust M1 or M2/grating parameters and then repeat M3 adjustments to get better flux. Ensure that the beam remains homogeneous during these adjustments. The M2/grating angles can be changed at fixed CFF without changing the EPU gap by changing the energy in PyDM. This step usually occurs prior to energy calibration, so obtaining maximum signal is helpful for getting more accurate energy calibration.]
In Phoebus, camera images can be processed in several ways.
Firstly, hit "CONNECT" and "Start" to start acquiring images.
Image Mode can be set to continuous to view a live image.
The exposure time can be increased to get a stronger signal. Keep the exposure time sufficiently low such that the image does not become saturated.
It can be good to start with the "Autoscale (N Sigma) box checked when searching for the image. After the image is found, however, this box can be unchecked to reduce the static in the background. The maximum and minimum intensities can be manually set such that they capture the range of the actual signal, and the image intensity appears stronger.
The physical camera lenses can be adjusted/twisted to tune the focus.
If the camera is on, connected, and started acquiring images, but an image does not show, it might be necessary to Reboot IOC and then start acquiring again.
RSoXS PGM Alignment
(Copied from Eliot's notes) The current procedure is mostly automated:
1.) Put HOPG (or another standard with a well understood prominent resonant TEY peak at a known energy - here we will use the sharp exciton resonance in HOPG at 291.65 eV) into the beam - (note that this involves basic alignment to ensure some beam getting down into the chamber - for that please see beamline alignment)
2.) run the tune_pgm command with a selection of cffs (constants of fixed focus). These choices will vary depending on the grating. Generally for the 250l/mm gratings, choices between 1.3 and 1.6 are ok and for 1200 l/mm, between 1.6 and 1.8 are ok, although if the resolution is every too bad to resolve the resonance we are looking for - as is the case at the extreme edges of the ranges - clearly as the highest intensity peak, a different cff should be chosen)
this function will use whatever the current offsets of the grating and mirror are and scan the grating through a range of angles which should contain 291.65 eV, For each of the cff conditions, the locations of the measured peaks are gathered. Finally all of the grating positions and mirror positions which produced the peaks are put into a optimization code and the offset of both motors is found which best fit those sets of angles.
3.) Monitor the grating peaks, and if in all cases the highest intensity is the peak of interest, check that the offsets are reasonable, and accept the fit results - the offset values will be set
4.) check that the operation was successful buy running a scan over the test sample and ensure the peak of interest is where is should be. If not, repeat the above steps.
General PGM operation - use this ONLY if the above does not work or to commission a new grating.
It is almost always safe to move the mirror in the negative direction, and the grating in the positive direction.
If you want to move both the grating and the mirror in the negative direction, move the mirror first
If you want to move the grating and mirror in the positive direction, move the grating first
be aware that if at ANY point the grating and the mirror match angles, zero order light will propogate down the beamline - essentially turning the grating into a mirror. The intensity of photons down the beamline will not be monochromatic, and will include all photons that can bounce off of a mirror at those angles. At high angles this is safe and useful for alignment, however at low angles the intensity of light can be quite high, so any sensitive downstream equipment should be retracted.
Homing PGM
Note that with the absolute encoders which have been in place since May 2022, this procedure is no longer necessary, but I leave it here for reference.
Move Mirror2 pitch to -11.5, then make 1 deg step negative (speed 0.14 accel .1 second)
Mirror2 at negative limit: -11.7545(3) @ gold #2
Set whatever offset necessary to make the readback equal to the above value
Move grating pitch to -14.5, then make 1 deg step negative (speed 0.14 accel .1 second)
Grating at negative limit: -14.63955(30) (1200l/mm grating, others need small)
Set whatever offset necessary to make the readback equal to the above value
If far away from correct, set it, and verify and refine the location (procedure below), repeat the above steps several times to be sure.
Verify homing is correct
scan a known peak (e.g. HOPG normal incidence at 291.65)
if within ~.5 eV
Go to HOPG peak at 291.65, refine offsets manually
Set mirror2 offset to above value +.001
Set grating offset to above value +.001
Set energy to 291.65 again
Check intensity
If higher continue both simultaneously in that direction
If lower go in opposite direction
try to get the joint offset to within 0.001 deg
If further off
set mirror2 offset to current value +.012
set grating offset to current value +.012
scan again
If direction of move was correct, repeat until within ~.5 eV then use peak method
Repeat known spectrum
check by changing the cff over a relatively large range, this should ONLY change the resolution, there should be NO shifting of energy
if there is, the grating and mirror are out of sync, and zero order scans are required.
if accurate, repeat steps 1 and 3 from the homing routine, and record the new values in this document
(if this is drifting over time, the limit switches may be unreliable)
To change grating
Go to a well known standard spectrum with a prominent peak (I use HOPG at normal incidence 291.65 peak)
Offsets for each grating are in the grating parameters and for the mirror in the mirror parameters CSS screens - these are set automatically anytime the change grating routine is run.
Run the tune PGM procedure above. If for some reason automatic alignment is not working the following steps should work.
Set the energy of the peak (291.65) as usual. Note the grating pitch value
Scan the grating pitch motor very very small moves to find the peak value. This can be very tricky, do NOT move the mirror during this process.
If there is a clear peak, go to that peak location.
Set the grating offset so the current location is the pitch value noted in step 2.)
Do an energy scan and ensure the spectrum looks normal. Note the CFF will likely have to change quite a bit, and the resolution might be quite bad until you get it right. - this is the reason for the automatted procedure which optimized the cff as well is needed. doing this by hand is not reasonable.
To change mirror stripe
Follow the grating steps above except with the mirror2 pitch rather than the grating, except the CFF should not have to change, and the offsets should be VERY small for the mirror motor
To do a zero order scan:
Only do this if you are completely lost and none of the previous settings are making sense
The purpose of this is to realign the mirror and grating pitch relative to each other.
I would do this on the gold grating, and the 1200 l/mm grating.
Close the RSoXS slits 2 DO NOT DO THIS IN ANY DOWNSTREAM LOCATION WHEN THE RSOXS SLITS ARE OPEN
Insert the closed fast shutter,
Insert the RSoXS Izero Mesh,
Home and close the exit slit to nominal small opening (25 microns).
You will be monitoring the Izero mesh current, so open that up in a way that you can watch while completing the rest of the steps
Set the mirror pitch to -11.5, then move the negative limit.
Set the grating offset to -14.5, then move to the negative limit.
From the limit, move the grating 5.64 degrees positive. This should put you at ~ -9 degrees.
Scan the mirror pitch from 2-3 degrees relative to the negative limit slowly with .01 degree steps.
At some point there should be a very narrow peak, note the position
Relative to that peak location scan from +.02 to -.02 degrees at .0005 degree steps.
Once scan is done, close the shutter (PSH4)
Note the peak location.
With the upstream shutter closed, move the mirror pitch to the peak location
Set the mirror2 pitch offset so that the mirror and grating position readbacks are identical. (No motors should have moved)
Move the grating to -6 degrees, open the upstream shutter again
Scan the mirror from -6.02 to -5.98, ensure peak is at 6 degrees, if not, refine the mirror offsets until it is.
If there is no peak in this range, increase the range to 0.1 degrees and try again - it is likely that there is a encoder problem with the PGM
Correct position (repeat steps 9-11)
Set a nominal energy, to move the mirror and grating away from this position
Open the upstream shutter
Check and refine alignment with the "verify homing is correct" procedure
Venting and pumping down main chamber
Venting
Isolate the main chamber by closing PSH10, GV27, GV27A, GV28, and GV29. The intention is to close two upstream and two downstream valves.
Unscrew the 24 V cables for the GV27A and GV28, and unplug them. Tag the cables with a piece of tape labelled "Do not plug in". This step will prevent unintentional opening of the gate valves by remote users and unintentional plugging in of the cables by onsite staff.Hardware for G27A motor. To disconnect the electrical connection, loosen the screw in the red box, and unplug that unit.
Isolate the turbo pumps for the main chamber, and turn off the main chamber ion gauge.Orange boxes: valves to isolate turbo pumps to the main chamber. Purple box: turns off ion gauge for main chamber.
Turn off all internal components and warm up any cooled items (i.e. detectors) well ahead of venting.
If needed, unmount any hardware component from the main chamber. Venting safety training is required to proceed. Before anything is removed from the chamber however, the Lead Beamline Scientist MUST be consulted and potentially radiation safety staff of NSLS-II. Any flange that is removed and not replaced or replaced by anything else, even an identical flange must be approved ahead of time. To unmount hardware, use standard UHV CF flange practice - consult lead beamline and vacuum techs if there is any possible confusion. Venting large vacuum chambers can be highly dangerous and if done in the wrong sequence, or if the chamber is under more or less pressure than expected life threatening situations can occur. In almost all cases contact the lead beamline scientist ahead of time, and only perform actions with them present. Slightly loosen all bolts on the relevant flange bit by bit working your way around the flange, and continue around until all are finger loose. Then, depending on if the item will fall off without support, remove all bolts, or leave enough bolts to ensure it is minimally secured before. Finally, carefully remove the final two bolts and unmount the hardware. This may require multiple people to handle moderately heavy components (a beamstop assembly), or a crane or riggers to remove any of the large flanges (i.e. the manipulators, the load locks, the detectors etc). Before removing the item prepare yourself visually and check that any internal delicate parts do not hit the port or walls while the item is carefully removed.
If any component is unmounted from the chamber, either cap the port with an equivalent blank flange and brand new copper gasket and pump down immediately (if it will be more than a few hours), or cover with aluminum (if less than a few hours) the longer the chamber is vented to atmosphere the harder it will be to pump down to pressure again. Ensure that the venting groove is lined up with the main chamber port. As commonly done with other bolted assemblies (e.g., car tires), gradually tighten opposite pairs of bolts in a circular fashion such that there is even pressure on the copper gasket throughout the tightening process.
Pumping down
Essentially, follow the venting protocol in reverse.
Ensure that all flanges on the main chamber are mounted and secured.
After the main chamber achieves sufficient vacuum, plug in beamline gate valve electrical connections, and open the gate valves.
Because the vacuum switch on the chamber will have tripped, the Flocos will need to be informed for the beamline to be reenabled. The Lead Beamline scientist will usually take care of this, as the flocos will want their assurance that the shielding did not change.
Shutting off and turning on motor controllers/IOCs
Shut off a controller and disconnect motor
Note current motor positions in PyDM (e.g., post-beam time motor positions after running nmode) to ensure that they remain the same when the controller is turned on later.
On the RSoXS control computer, open a new tab in the terminal app and log in. Then log into an IOC (e.g. ioc1) by entering ssh xf07id-ioc1. Alternatively, log in directly to the ioc following this example:
su pketkar@xf07id-ioc1
The RSoXS hardware may be connected to IOCs 1, 2, 3, and 7; primarily 1 and 3 are used. IOC 1 runs the sample manipulation motors, temperature controller, and I400 monitor among other items. IOC 3 runs the cameras, quadEM monitors, and slit motors, among other items. The IOCs (Input/Output Controllers) are like the "brains" of the motor, communicating instructions between the control system and various devices for how to move. The IOCs also collects data from encoders, sensors, etc. and can use it as feedback for controlling devices. The motor controllers are like the "braun", enabling the actual movement.
Identify if the desired controller is in this IOC by running:
manage-iocs status
List of controllers on selected IOCAdditionally or alternatively, identify which controller needs to be shut off by going to phoebus and finding the controller name that corresponds to the desired motor.e.g., the MC21 controller corresponds to the RSoXS UpDown, RSoXS Rotation, etc. motors
Turn off the desired controller (e.g., MC21) by running:
dzdo manage-iocs stop MC21
The controller name after stop should not include the "softioc-" and ".service" that are displayed after running manage-iocs status.
Check that the controller has been turned off by running manage-iocs status. The status of the desired controller should read "Stopped" instead of "Running".
Physically switch off the desired controller, which is located in the cabinet on the SST2 side of the beamline.Controller cabinet is on SST2 side of the beamline. Controller names (blue box) and associated motors (orange box) are labelled. Green box shows power switch for a controller.
If needed, disconnect wire for motor that will be serviced (e.g., x motor).Wire that connects motor (e.g., x motor) to controllers
On the RSoXS control computer log in on the terminal app, access the desired IOC (e.g., ssh xf07id-ioc1), and turn on the desired controller (e.g., dzdo manage-iocs start MC21).
Run manage-iocs status, and check that the controller status shows "Running". Also check on phoebus that motor statuses are enabled and on PyDM that the motors are not grayed out.
Perform homing for all motors. Generally, homing involves the motors moving to a physical limit as a reference position, and then moving back to the default programmed position. However, if a motor is already at a physical limit (e.g., y motor after sample bar removed from load lock), the equipment is set up such that it cannot be moved to the physical limit of the motor (e.g., beamstops), etc., other means of homing are required.
If any of the motor positions (e.g., RSoXS Up Down) in PyDM are not the same as they were before turning the controller off, click on "details" and correct the user offset (see details below). Then restart the IOC (e.g., dzdo manage-iocs restart MC21).Correct the user offset (green box), which can be accessed by clicking on the "details" button (red box) in PyDM.
For a linear motion motor, such as the RSoXS Up Down, User Offset = (current position in PyDM) + (prior position in PyDM before controller shutoff) - (prior User Offset).
For a rotation motor, such as RSoXS Rotation, first set the motor position to something larger than 360 (e.g., 362). Then, User Offset = (prior User Offset) - 360. The system is set up such that any angle larger than 360 degrees will restart as that angle minus 360.
If needed, use the channel archiver to find the previous positions.
Home the following motors by pressing "Home" for the associated motors in PyDM (or phoebus). The status should change from "Not Homed" to "Homed". These following motors have incremental encoders, and the position might be lost during a power shutdown.
WAXS (and SAXS, if connected) detector camera (MC21 axis 5 and 6)
Each pair of slits, 6 pairs total (MC19 axis 1,3 and MC20 axes 1,3,5,7). The slits will home in pairs to ensure that one slit does not collide into a screw of an opposite slit.
RSoXS fast shutter translation (MC19 axis 6)
RSoXS Izero screen translation (MC19 axis 5)
Note: the beamstop translations (MC21 axes 7,8) should technically also be homed. However, the angle that they are in the port aligner makes it difficult to reach the limit switch, so this will fail. Thus, it is better so it is better to NOT home them and instead, check their position the first time they are used. Be extremely careful, and do not assume they are blocking beam; attenuate the beam and take very low-time exposures until it is ensured that the beamstop is in the correct position.
To home and autodisable motors that cannot be reset in PyDM, open the application to reset motors by going to phoebus and opening (File --> Open) /opt/css/opi/production/cs-studio-xf/07id/common/motor/asyn_launch.bob. This is used for motors with absolute encoders (i.e., absolute position is always known, axes 1, 2, 3, 4). These motors are essentially always homed, but they might show up as "Not Homed" on PyDM. This is because FMB has attached homing routines that are intended to run motors to their limit and then look for a home, which is not there; thus, pressing the "Home" button on PyDM will fail. The asynOctet will set the software switch to "home" these absolute encoder motors.Open asynOctet for the desired controller (red box) and use the ASCII input (green box) to reset motor positions and homing. This gives commands to the motor controller.
If the absolute encoder motors are not showing up as homed, run the following:
In the ASCII input, enter M1x16, where x goes from 1 up to 4, and hit enter for each. x represents the axis with axis 1 representing the x motor (RSoXS Inboard Outboard) on the MC21 controller, axis 2 = y motor (RSoXS Up Down), axis 3 = z motor (RSoXS Upstream Downstream), and axis 4 = RSoXS Rotation.
The value should read out 4, which means that the motor is homed. If the value is not 4 (might be 0 after restarting), enter M1x16 = 4, which sets the value equal to 4.
Double check both in asynOctet and PyDM that these motors show up as "Homed".
If motors to not automatically disable themselves after a move, use asynOctet to autodisable motors so that the power to the motor is turned off after the motor move is complete. Do this for all of the motors on the RSoXS controllers (MC19,20,21). This reduces ampfault occurrences. This is a software-based disabling, whereas the process for clearing ampfault errors during beamline operation is hardware-based disabling.
In the ASCII input, enter px05 = 1, where x goes from 1 up to 8, and hit enter for each motor that is not autodisabling.
These motors should show as disabled in PyDM.
If the RSoXS_Sample_Imager is not homed, use the MC19 asynOctet, and enter i724=$420401. There are no limit switches, but the motor controller forgets this, and panics seeing them both set at once; thus, the motor cannot be used.
If any motor is not being used (e.g., the SAXS camera detector motors on April 2024), disable it by running ix00=0 in asynOctet (x=6 for the SAXS camera detector on MC21). This will make the controller completely ignore any command sent to that axis.
After homing and autodisabling, ensure that all motor positions are where they are desired. For example, homing may leave slits positioned in the beam path. It is easiest to run nmode and check that all RSoXS components are clear.
If needed, general motor control can be performed on phoebus.
Access motor controller interfaces on phoebus by going to File --> Open --> opt/css/opi/production/cs-studio-xf/07id/edl_conv, navigating to the folder the corresponds to the motor controller of the desired motor, and then opening the .bob file for the desired motor. If unsure which motor controller corresponds to the desired motor, it can be helpful to open the edl_conv folder in file explorer, and then search for the name of the desired unit (e.g., "DM07" for diagnostic module 7) and then find the file location.
The motor name can be obtained by hovering above the motion dimension.Phoebus interface for Mirror M4CD. The motor name (red box) can be obtained by hovering above the motion dimension (e.g., green box)
Other motor controller troubleshooting
If a motor controller needs to be reset without a complete power cycle, the enable/disable cable coming out the back of the controller can be disconnected and reconnected (the cable is screwed onto the controller). This way, the motors do not have to undergo homing. This enabling/disabling also can be done via software using the DIODE box.
Hardware lubrication
In general, lubrication should be performed every 1.5 years and more frequently if a component experiences greater stresses.
Y (Up Down) leadscrew lubrication
Schedule with the techs.
Ensure the following:
The motor controller is on. Otherwise, manual handcrank will need to be used.
Downstream stations are not being used.
Solid sample load lock is pumped down to sufficient pressure and manual gate valve is open.
Start with the y motor at its top limit. Techs will lubricate the bottom portion of the leadscrew.
Then move the y motor to its bottom limit. Techs will lubricate the top portion of leadscrew.
Return y motor to its top limit, and techs will relubricate the bottom portion of the leadscrew.
X micrometer lubrication
Schedule with the techs.
Ensure the following:
All hardware is moved out of the way of the beam path.
Motor controller is off.
Backup micrometers are available in case the current one breaks during dis/reassembly.
Techs will detach motor stage, take apart the micrometer, and rebuild after lubricating.
Optimizing RSoXS slit positions
Note the current settings.
Slit, beamstop, etc. positions can be found in the Bluesky codebase (e.g., see WAXSNEXAFS). This is helpful in case it is necessary to reset to the current default positions at any point, and it can be useful to compare how slit positions change over time.
Note down the current flux. It is a good practice to keep a record of how the flux changes with respect to slit settings.
Ensure that all CCD cameras are retracted, and only the photodiode is in the beam path (e.g., the WAXSNEXAFS configuration). Move the sample bar out of the way (RE(load_samp("diode"))).
Set the energy to 270 eV (e 270), open the fast shutter, and note down the starting signal values on the photodiode and Au mesh. Close the fast shutter after this.
Beamstop_WAXS.read()
Izero_Mesh.read()
These signal values also can be viewed in the live plot in Phoebus.
Capture a single image using the current slit settings to gauge how much slit scattering appears on the camera.
Refine and set the energy resolution using the Slit 1 vertical aperture.
Ensure that the camera detector is fully retracted and that only the photodiode is in the beam path.
Open Slit 3, Slit 2, and horizontal aperture of Slit 1 all the way.
The numbers correspond to the aperture size (in mm) of the prior set of slits, in which hsize and vsize correspond to the horizontal and vertical apertures, respectively. Note that opening slits this widely can make the beam very large, and it is not guaranteed that the fast shutter would block the full beam from hitting sample. Load important/sensitive samples closer to the top of the bar to protect from beam damage.
Load the HOPG reference: RE(load_samp("HOPG"))
Set the Slit 1 vertical aperture to a desired value (e.g., RE(bps.mv(slits1.vsize, 0.025))). Then run a carbon edge TEY NEXAFS scan (RE(do_nexafs(edge='carbon'))). Repeat at different Slit 1 vertical aperture sizes. Aim to maximize flux (larger aperture size) while keeping high energy resolution (lower aperture size).
Choose the largest slits1.vsize before the HOPG peaks broaden. It can be helpful to overlay the HOPG scans at different slits1.vsize values, normalize to the post-edge value to remove flux effects, and compare the 290-295 eV region, where the impacts of the aperture size on energy resolution are most prominent.
Set the Slit 1 horizontal center and aperture size.
Move the sample bar out of the beam path: RE(load_samp("diode"))
Set a moderate aperture size (RE(bps.mv(slits1.hsize, 0.1))), open the fast shutter (it does not open on its own by just running the scan command), and then run a scan across different center positions. Remember to close the fast shutter at the end of the scan.
This runs a scan where the center of the Slit 1 horizontal aperture is moved from a position of -1.5 mm to 1.5 mm, and 51 data points are collected on the WAXS beamstop and Izero mesh. The aperture size stays the same throughout the scan. Note, a relative scan also can be performed using bp.rel_scan; in this case, the position inputs are relative to the current position with 0 corresponding to the current position.
Adjust the scan parameters or aperture size as needed so that a maximum can be seen in the slit scan. Pick the aperture position value that gives the maximum flux for both the beamstop and I0: e.g., RE(bps.mv(slits1.hcenter, 0.7)).
Then run a scan across different slits1.hsize values: e.g., RE(bp.scan([Beamstop_WAXS, Izero_Mesh], slits1.hsize, -1, 3, 51)). Adjust the scan parameters as needed to see the full range of fluxes between the slits being fully closed and fully out of the beam path as well as the flux plateaus at both ends. Note that an aperture size of 0 may not necessarily correspond to the actual motor positions that fully block the beam.
Note the slits1.hsize value at which the WAXS beamstop signal reaches a maximum plateau; this is the maximum limit for slits1.hsize. Larger values indicate that the beam size might be larger than the WAXS photodiode.
For solid sample configurations, choose a slits1.hsize value that is well below the maximum limit to avoid direct beam hitting the camera detector and minimize sample damage that could result from too large of a flux. For liquid sample configurations, set the slits1.hsize somewhere close to or slightly below the maximum limit to maximize flux on these samples. E.g., RE(bps.mv(slits1.hsize, 0.05)).
Set the centers and determine the minimum aperture sizes for Slit 2.
With the sample bar still out of the beam path and Slit 1 parameters set based on the above scans, set the Slit 2 apertures to moderate sizes: e.g., RE(bps.mv(slits2.vsize, 0.1, slits2.hsize, 0.1)).
Scan and set the vertical and horizontal center locations: e.g., RE(bp.scan([Beamstop_WAXS], slits2.vcenter, -1.5, 1.5, 51)), RE(bps.mv(slits2.vcenter, 0.7)). In this case, it is not necessary to collect data at the Izero_Mesh because Slit 2 (and Slit 3) is downstream of the diagnostic module.
Scan the vertical and horizontal aperture sizes: e.g., RE(bp.scan([Beamstop_WAXS], slits2.vsize, -1, 3, 51)).
Note the slits2.vsize and slits2.hsize values at which the WAXS beamstop signal reaches a maximum plateau; these are the minimum limits. Lower values indicate that the slits might start blocking a portion of the direct beam, and this should be avoided.
Set the vertical and horizontal aperture sizes to values slightly larger than the minimum limit. At this point, the purpose is not to find the exact aperture size, but to get a rough sense of the minimum limit. If the Slit 1 aperture sizes ever change past this point (e.g., the beam size might need to be reduced further to avoid having direct beam on the detector), it is good to rescan Slit 2 and get an updated set of minimum limits.
Scan and set the Slit 3 vertical and horizontal center locations after setting the aperture sizes to 0.1: e.g., RE(bp.scan([Beamstop_WAXS], slits3.vcenter, -1.5, 1.5, 51)), RE(bps.mv(slits3.vcenter, 0.7)). Aperture sizes can be set to approximately double the sizes that are used for slits 2, and they will be refined further using the CCD camera.
Refine slits1.hsize, slits2.hsize, slits2.vsize, slits3.hsize, and slits3.vsize using the CCD camera.
Ensure fast shutter is closed, and that the camera detector is protected from direct beam.
Bring the WAXS CCD camera into the beam path: e.g., RE(load_configuration("WAXS")). Note, this will undo all the slit settings from earlier, so these will have to be entered again. Alternatively, only the camera can be moved into the beam path, or a temporary configuration can be set in the Bluesky codebase with a running set of slit parameters.
For the first scan as well as subsequent scans in which settings are changed significantly, start with a low exposure time (exp 0.01) to ensure that direct beam is not hitting the camera.
Systematically and in small increments, vary the aperture sizes (e.g., RE(bps.mv(slits3.hsize, 2))), and then capture single images. It is useful to note the flux values (WAXS beamstop signal) during each iteration of slit settings to ensure Slits 2 and 3 do not impede any portion of the direct beam. Often it is helpful to start with Slit 3 (aperture should be large enough to avoid long range scattering streaks but small enough to reduce signal closer to the beam stop) and then make minor adjustments to Slit 2 as needed, keeping in mind the minimum limit. Occasionally, it may be necessary to change slits1.hsize and then redo refinement of Slits 3 and 2.
Images can be captured at different energies to get a better idea of how slit scattering is impacted by the energy and flux.
Note down the aperture sizes that give the best images (e.g., least slit scattering, minimal artifacts).
Optimize the beamstop position.
As described earlier, the beamstop can be withdrawn into and out of the beam path using a motor (PyDM) or rotated slightly in the opposite direction using the manual port aligner. If the beamstop position adjustment is any larger than the most minute movement, it is safer to retract the CCD camera, ensure that the beam is hitting the beamstop photodiode, and only after that, bring the camera detector back into position.
Very important: after ANY adjustments to the beamstop position, reduce the exposure time (exp 0.01) before capturing an image to ensure that direct beam is not hitting the camera detector. After this, the exposure time can be increased carefully for subsequent scans at the same beamstop settings.
This can be done with the sample bar out of the way, or it can be helpful to load a strongly scattering reference sample, which makes it easier to gauge how the beamstop should be moved to place it in the center of the scattering.
Update instrument configuration in the Bluesky codebase with the newly optimized slit parameters and beamstop position. Restart Bluesky for these updates to take effect.
For reference, collect images of the open beam.##Very important: attenuate the beam intensity by reducing the aperture size of Precision Slit C (Slt:11 - YGap in PyDM) from -3.05 (default) to -0.01. Also, reduce the exposure time (exp 0.01). Do not run open beam measurements with no beamstop at any larger aperture size and exposure time.
Withdraw the WAXS beamstop in PyDM to a position of 3.
Reduce the binning to get higher resolution and less likelihood of saturation.
binning 1
Capture an image with all slits (RE(bps.mv(slits3.hsize, 10, slits3.vsize, 10, slits2.hsize, 10, slits2.vsize, 10, slits1.hsize, 10, slits1.vsize, 10))) and gold mesh retracted to characterize the beam itself.
Capture an image with Slits 1, 2, and 3 in their newly optimized positions if desired.
Images can be captured at different energies if needed.
After the desired images are captured, move the beamstop back into the beam path, increase the Precision Slit C aperture size to -3.05 (and move other slits to their optimized positions), and increase binning (binning 4).
Resetting and setting upphotodiode readout
I400s
If needed, restart the IOC. Log in on a new terminal tab, log into IOC7 (ssh xf07id-ioc7), and run dzdo manage-iocs restart RSoXS_I400. On Phoebus, it may be necessary to click "Enable" in the "Control" --> "Automatic Acquisition" menu to start reading signal.
Use Bluesky to run the following:
RE(setup_diode_i400())
RE(setup_slit1_i400())
RE(reset_diodes()) (if the electrometer is already powered on and just needs a reset)
If the slit1 I400 needs power cycling, disconnect and reconnect its power cable physically.
The readout can be viewed on phoebus.I400 readout can be viewed and adjusted in phoebus.
Test the photodiode by turning on and off the chamber light, and check that the signal increases and decreases correspondingly in the RSoXS_Currents_archiver live plot on phoebus.
The parameters can be adjusted tune the gain, noise, etc.
Integration time: the amount of time signal is collected for a single raw data point
Trigger points: the number of raw data points that are averaged to obtain a single data point in exported data. Larger integration times and more trigger points results in lower noise, but reduces data collection frequency.
Active capacitor: lower capacitance results in higher gain but lower saturation threshold. Saturation threshold = capacitance/(integration time)
QuadEMs
Open readout in phoebus. The following parameters should be set as such:QuadEM readout can be viewed and adjusted in phoebus.
Integration time: 0.0004
Ping Pong: "Both"
Range: 50 pC. Note that 12 pC should work better in theory, but has not worked well in practice.
Averaging time: 2.00E-1
Click the "Done" and "Acquire" buttons, and it should start updating. If not, restart the QuadEM softIOC (log in on the terminal app, ssh xf07id-ioc3, dzdo manage-iocs restart quadEM-1). In some cases, it may be necessary to to disconnect power from the +9V power port for a few minutes, reconnect power, and then restart the softIOC.
If the beam is off, zero the values by increasing the Averaging time to 4 seconds, waiting for a few readings, clicking the "Compute" buttons under each channel, and watching the output in the RSoXS_Currents_archiver live plot on phoebus until the signals are suitably noisy around zero on average. Then setting the Averaging time back to 0.2 seconds. The 4 second averaging time provides better signal statistics, which allows a more accurate offset to be subtracted from the signal.
Greateyes CCD camera
Pre-installation benchtop test
Before installing the camera into the chamber, it is a good idea to test that the camera can capture dark images properly on the benchtop.
Ensure the camera is secured into its metal cylinder so that no ambient light reaches the sensor.
Connect the camera to its control unit using a 50-pin cable, and connect the camera to a Windows computer using an ethernet cable. Connect the control unit to electrical power. Switch on the control unit, and ensure that the LED light on the camera electronics turns on.Greateyes camera ethernet and 50-pin connections during benchtop testing.
Connect the windows computer to the camera's IP address. The correct IP address can be found using the command prompt and pinging the camera server (e.g., xf07id-det1.nsls2.bnl.local).Connecting to Greateyes camera's IP address via ethernet.
Ensure the camera server is connected by entering the IP address onto a web browser, using the same Greateyes login as normal, going to the System tab, and checking that the Camera server shows green.
Download and open the greateyesVision.exe software and connect to the camera.Connect to the camera by first going to Options and then selecting Ethernet in the Camera Interface section (red box). Then, go to the Camera Setup tab, type the IP address for the camera, hit the enter key so that the box shows as cyan, and then click Connect. The red light should turn green after the camera is connected.
In the Camera Setup tab in the Temperature Controller section, Cam1 can be set to the desired temperature. On the benchtop, it is possible to cool the camera moderately (down to -20 °C) using air alone; further cooling requires the camera to be connected to water. In the Readout Setup section, the Frequency can be set to 4 amps so that all 4 quadrants of the camera are read out.
In the Acquisition tab, click on the Single Shot button to capture one image. The exposure time and other capture parameters can be adjusted. The resulting image and intensity line cuts can be viewed to check if the image is reasonable.
Turn on the camera (routine operation)
Switch on the camera controller.Switch off camera.
Open a Google Chrome window, and go to xf07id-det2.nsls2.bnl.local. Log in using Username: admin, Password, greateyes. Click on "System" on the left-hand panel. Click "Start" on any "System Status" item that has a red light in front of it, and then click "Refresh"; this may need to be repeated several times until all lights show green.Greateyes system statusThis is the lowest level control of the hardware. In some cases, the Analog Power might appear to be green, but it is not actually on. If hitting the Stop button on Analog Power does not turn it off, the Analog Power was not on to begin with. In this case, switch off the camera for a few minutes and retry turning it on.
The WAXS detector is located on IOC 3. The IOCs are the servers that run all of the EPICS code for the hardware. The Bluesky commands call on this EPICS code.
Start the camera sot IOC:
dzdo manage-iocs start greateyesCCD2
If this was done correctly, you should see the WAXS detector line in the pydm window on the first display flash grey, then purple and finally update with a correct temperature again, with temperature control “disabled”. Change it back to enabled (and set the target temperature back to -80 °C, if it has changed) as soon as you can, so the camera does not warm up too much.WAXS camera temperature control
Check that the sensor is working by capturing a dummy image.
It may be necessary to turn_off_checks() if the camera is retracted, there is no beam, or if another station is running. In this case, also go to DIODE-MTO:1 in Phoebus, and hit Disable on the input enable mask; as a result, the blue square should light up blue, but the corresponding red circle should not light up.
Set the desired exposure time in Bluesky (e.g., exp 0.01)
Capture an image:
RE(bp.count([waxs_det], num=1))
If the sensor is working, the image should show some artifacts, such as bad pixels or distinct quadrants, even if it is a blank image. If the sensor is warm, then there will be a more visible intensity gradient in the image. If the sensor is not working, then the image will homogeneously show low intensity, and the camera should be restarted.
Check that the chamber light is not on when capturing an image. One of the signs may be that 4 quadrants are visible in the image. After the chamber light is shut off, wait at least 30 s (sometimes even longer) before capturing an image. To ensure a new dark image is taken, type dark_... and tab complete to clear the old dark image.
Wait for the camera temperature to cool below -77 °C before starting scans on samples.
Remember to turn_on_checks() or restart Bluesky before proceeding.
Switch off the camera controller. Switching off the camera controller will cause the camera to warm up, which is fine for occasional and long shutdowns. On a more frequent basis, however, it is better to switch off the camera for 5 min or less and then turn it back on to avoid the sensor warming up too much.
Store away camera
Ensure that the wire-protection metal cover is secured onto the camera. This protects the wires that connect the sensor to the electronics and should already be on when the camera is installed in the RSoXS chamber.Error creating thumbnail: File with dimensions greater than 12.5 MPGreateyes camera wire-protection cover shown from 4 side views
Put on the metal cap in front of the sensor chip to protect the sensor.Greateyes camera sensor protection cover used when storing and shipping the camera.
Place the camera flange onto the metal cylinder such that the sensor faces the inside of the cylinder, and the electronics face out from the cylinder. Ensure that the leak grooves on the flange align with the grooves in the cylinder. Secure the flange using bolts and washers, being careful to not touch the knife edge, which is facing outside.Error creating thumbnail: File with dimensions greater than 12.5 MPGreateyes camera stored into metal cylinder with sensor facing inside the cylinder.
Place this setup into the Pelican case with the cylinder facing downwards. Ensure all sides have sufficient cushioning before closing the Pelican case.
Reference scans
Below is a compiled list of reference scans to monitor the state of the beam and beamline hardware.
Dark signals should be monitor when RSoXS does not have beam.
Several open-beam energy NEXAFS scans (over various edges) over a duration of time to monitor evolution of beam intensity, signal quality, normalized signal, etc. Ideally, monitored parameters should include (WAXS) photodiode, I0 mesh, RSoXS slit1 drain currents, mirror temperatures, EPU60 offset parameters, etc. The time evolution at any energy can be extracted from from a series of scans.
If the energy is not calibrated, a time scan can be performed using bp.count().
Several HOPG energy NEXAFS scans across the carbon K-edge over a duration of time. This can serve a similar purpose as the open-beam energy scans and also ensure that the energy calibration is not shifting by checking that the HOPG peak positions remain at the same energy throughout this duration.
Time scans on the various fluorescence screens throughout the beamline to evaluate beam intensity and motion. These fluorescence screens also can be scanned across various beamline parameters (energy, EPU gap/phase, mirror settings, etc.)
Several RSoXS scans on a reference material (e.g., SBA-15) over a duration of time to assess beam motion if the scattering pattern is moving. RSoXS scans over a wide energy range also can serve as another way to calibrate energy based on the peak q position.
Exposure time series on CCD camera to assess dark and read noise.
Energy scans on blank SiN window or other reference materials to assess spot-to-spot variability. Energy scans on the same spot over time can help assess beam damage.
Documenting work/updates on beamline
Tasks involving hardware, software, or accounts/admin, especially those that involve multiple BNL staff, can be entered into jira.nsls2.bnl.gov. Tickets can be entered and assigned to staff as needed. Typically, a more informal discussion is first done on the nsls2 Slack, after which a Jira ticket can be created to document the important information and discussed solutions.
