Lead Data Engineer

  • Microsoft
  • New York, NY, USA
  • May 22, 2020

Job Description

Lead Data Engineer, PromoteIQ Microsoft Advertising PromoteIQ provides intelligent vendor marketing solutions for the next generation of e-commerce. Our platform helps retailers implement, automate, and scale their brand-funded digital vendor marketing programs. We sit at the intersection of marketing and e-commerce and have a singular mission of empowering retailers and brands to maximize their e-commerce performance. PromoteIQ embodies a strong startup culture that values diversity, collaboration and craftsmanship - and above all else, results. Our bias towards execution balances critical thinking, root analysis and pragmatic problem solving. We expect a lot from one another and value our thoughtful and intellectually curious company culture. PromoteIQ is headquartered in New York City and supports a global footprint of e-commerce retailers and brands. The company was acquired by Microsoft in Aug, 2019 and continues to operate as an independent division within Microsoft Advertising. Learn more at https://www.promoteiq.com. Microsoft Advertising is a worldwide Sales, Marketing and Services organization on the cutting edge of the digital advertising industry. Microsoft Advertising offers a compelling portfolio of advertising products, innovative solutions and the opportunity to engage with some of the brightest minds in the digital industry. Microsoft Advertising is the destination for experienced, collaborative, and passionate digital advertising professionals seeking a rewarding career and lifestyle. Who We’re Looking For At PromoteIQ, data plays an integral role in our product, and software engineers on our data engineering team build the pipelines that power reporting and analytics for our e-commerce promotions platform. The infrastructure and applications that you'll build on the data engineering team will have broad and critical reach in powering real-time auction decisions, becoming multipliers on our revenues, and forecasting supply and demand for our customers. Responsibilities Ship high-quality, well-tested, secure, and maintainable code Design, develop, and maintain data pipelines and back-end services for real-time decisioning, reporting, optimization, data collection, and related functions Manage automated unit and integration test suites Work collaboratively and communicate effectively with a small, motivated team of engineers and product managers Experiment with and recommend new technologies that simplify or improve PromoteIQ's stack Participate in an on-call rotation and work occasional off-hours Qualifications BS/MS in Computer Science or a related technical field One or more years of experience leading a team of engineers through technical architecture at all stages of the Software Development Lifecycle Two or more years of engineering experience with data pipelines and infrastructure Seeking candidates with 6+ years of experience in: Architecting, building, and maintaining end-to-end, high-throughput data systems and their supporting services Designing data systems that are secure, testable, and modular, particularly in Python, as well as their support infrastructure (shell scripts, job schedulers, message queues, etc.) Designing efficient data structures and database schemas Working with distributed systems architecture Incorporating data processing and workflow management tools into pipeline design (AWS EMR, Airflow, Kafka, etc.) Using profiling tools, debugging logs, performance metrics, and other data sources to make code- and application-level improvements Developing for continuous integration and automated deployments Utilizing a variety of data stores, including data warehouses (ideally Redshift), RDBMSes (ideally MySQL), in-memory caches (ideally Aerospike and Redis), and searchable document DBs (ideally Elasticseach) Wrangling large-scale data sets #PromoteIQ #MicrosoftAdvertising Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request via the Accommodation request form. Benefits/perks listed below may vary depending on the nature of your employment with Microsoft and the country where you work.