PCDS Keeps LCLS Users and Data Connected
"Light experiments don't usually produce a volume of data comparable to particle physics experiments," said Head of the Electronics & Software Research Engineering department, Gunther Haller. "But then, there's never been a photon experiment quite like this."
The Linac Coherent Light Source is unique in many ways, and SLAC's Photon Controls and Data Systems group knows it. The group plays a key role in ensuring the safe delivery of data from its birth in the experiment instruments, all the way to completed analyses. To do so, the PCDS group is constantly adapting to the challenges that arise from such a state-of-the-art and totally unique machine.
PCDS was initially formed as a core group of four people, Gunther Haller, Chris O'Grady, Amedeo Perazzo and Matthew Weaver, but it evolved in a larger group that now includes the equivalent of about 28 full-time staff, mostly from the Particle Physics and Astrophysics Directorate, and a few from the Office of the Chief Information Officer. There are five subgroups in PCDS: Data Acquisition, Controls, Information Systems and Networking, Data Analysis and Applications, and Safety and Infrastructure. The group is now a crucial part of the busy and bustling LCLS community.
The LCLS generates packets of high-energy photons to examine the atomic and molecular structure of matter. Those photon packets come from electron beams accelerated down the SLAC linac, and will eventually be distributed to six LCLS experimental hutches. All of the hutches have their own unique instruments, which may use a variety of devices to view matter in different ways. LCLS users also often bring custom made or one-of-a-kind devices of their own.
The PCDS group designed and built the LCLS controls and data system, which connects scientists and users to the instruments in the experimental hutches. The system carries and processes data captured by the instrument detectors to the control room. There, scientists can see and analyze the data, as well as remotely operate and monitor all of the devices in the experimental hutch. But to make this possible, the PCDS group must adapt the system to each instrument and device.
"Typically in high-energy physics you have many devices of the same kind that you build into one very large detector. [In the LCLS] the devices can be very different between instruments and also inside the same instrument," said Amedo Perazzo, the new LCLS department head for PCDS. "Scientists always want to employ the best and the latest, and the instrument is often not something that you've come across before. But it is important to modify the system quickly if you want to get the best science out."
PCDS provides fast processing and monitoring tools that allow ongoing evaluation of the data, giving the scientists the opportunity to review it in real time, and make changes, if necessary, to various experimental parameters like biases,, thresholds, and detector positions, so that they can get the most out of their beam time.
During the LCLS planning stages, participating scientists realized that users wouldn't have the data storage capacity to hold the enormous data sets they would generate. At most photon facilities, such as synchrotrons, they store their own data, often on laptops or external hard drives. But in its first run last fall, the LCLS generated up to 18 terabytes a day or 200 terabytes total, operating at only a fraction of its nominal data rate. That's too much data for users to capture or process using portable devices. So, the PCDS group members applied their knowledge from large particle physics experiments and built a database and data management system.
Once the data are safely in the database, PCDS provides the tools for scientists to make best use of it. The group designs the software required to analyze the data and teach them how to operate it, including accessing their data from their home institutions. Throughout LCLS construction and commissioning, Haller, Perazzo, and the PCDS team worked with LCLS scientists and instrument specialists to build the software, zeroing in on the best solutions as the LCLS project developed and the scientists better defined their needs and specifications.
"So we put it all in place, all the hardware and software, the infrastructure, the processors , the data acquisition, and controllers" said Haller. "And what the experimenters see is an interface in the control room that they can employ to analyze the data and monitor the experiment. And eventually we store the data, and the scientists can reprocess the data reanalyze it and have access to it when they leave SLAC."
In addition to building the controls and data systems, PCDS group members maintain those systems and respond to problems or calls for assistance. During the first LCLS user run last fall, PDCS members were on call all through the night and on weekends to troubleshoot any problems.
"The LCLS felt an enormous pressure to perform. It wasn't acceptable to lose beam time to a broken device or a piece of software," said LCLS instrument scientist John Bozek. "We would call [PCDS] at 3 a.m. and sometimes they could resolve the problem remotely and sometimes they would come in. They were extremely responsive. I can't say enough how pleased I was with their responses to our issues."
The PCDS group is busy preparing for the new science run coming up in May. The year will include modifications to the Atomic, Molecular and Optical science experiment, the commissioning of the Soft X-ray Science beamline with four new SXR endstation experiments, initial commissioning of X-ray Pump Probe instrument, the first of three LCLS Ultra-Fast Science Instruments, and modifications to the Front-End Enclosure X-ray Diagnostics instrumentation. The volume of data will increase with the start of the new experiments, as will the demand for on-call PCDS group members. Perazzo ran through the "to do" list for the group, then said with a laugh, "That's a lot."