Splunk Metrics and Monitoring Software Engineer

Perks and Benefits

• 100% Paid Healthcare
• 10% 401k in every paycheck
• 100% Fully Vested!

NOTE – Our positions require a Top Secret clearance, as well as the favorable completion of a polygraph. Applicants must be authorized to work in the U.S. We are unable to sponsor an employment Visa.

What You’ll Be Doing (We don’t love the bullet points, but we love the work!)

Build and sustain a distributed metrics and monitoring solution that helps ensure HPC infrastructure remains reliable, scalable, and observable. This role is centered on Splunk development, including SPL, dashboards, reports, data onboarding, field extractions, alerting, and event correlation. You will combine software engineering, Linux automation, log management, CI/CD, and infrastructure-as-code practices to deliver actionable visibility across high-performance computing environments.

Required Skills

• Experience using the Linux CLI and the Linux suite of tools
• Experience developing Shell/Perl scripts to automate manual processes
• Recent software development experience using Python
• Experience with Splunk SPL, dashboard creation, and report development
• Experience with Splunk data onboarding, field extractions, data models, and performance tuning
• Experience with Splunk alerting, incident management workflows, and event correlation
• Familiar interfacing with Log Management tools including Syslog-ng and rsyslog
• Familiar Message Brokers such as Kafka
• Experience with IaC (Infrastructure as Code) principles and automation tools including Ansible
• Experience with CI/CD principles, methodologies, and tools such as GitLab CI
• Experience with Git Source Control System

Desired Skills

• Experience with the Atlassian Tool Suite (JIRA, Confluence)
• Appreciates a sense of humor and the occasional well-timed joke.