Dit bericht verscheen eerder bij FOSSlife
Given the massive amount of data collected, managed, and consumed these days, data science roles — such as data engineering — are increasingly important. As Coursera explains, organizations need the right people and technology to ensure that data is in a highly usable state.
In this Careers article, we’ll provide an overview of the skills and responsibilities related to data engineering along with many resources to help you get started.
The Raw and the Cooked
“Data engineers design, build, and optimize systems for data collection, storage, access, and analytics at scale,” says CIO. They create data pipelines that convert raw data into usable formats, and they are “responsible for managing, optimizing, overseeing, and monitoring data retrieval, storage, and distribution throughout the organization.”
In preparing data for analytical or operational uses, data engineers “integrate, consolidate, and cleanse data and structure it for use,” says Ben Lutkevich. They also deal with both structured and unstructured data.
“Structured data is information that can be organized into a formatted repository like a database. Unstructured data — such as text, images, audio and video files — doesn’t conform to conventional data models. Data engineers must understand different approaches to data architecture and applications to handle both data types,” Lutkevich explains.
Skills and Responsibilities
The primary goal of data engineering is to make data available, accessible, and secure, says CIO. To do this, a data engineer’s toolkit includes skills and technologies related to:
Data engineers also need to be skilled in various programming languages, such as Java, Python, R, and SQL.
Common responsibilities for a data engineer include:
- Acquire appropriate data sets
- Clean, organize, and prepare data from various sources
- Develop, test, and maintain database pipeline architectures
- Automate manual data processes
- Ensure compliance with data governance and security policies
Courses and Certifications
Online courses can help you acquire data engineering skills; here are a few options to consider:
Popular data engineer certifications include:
You can learn more about data engineering from the following resources.
Other Resources
Dit bericht verscheen eerder bij FOSSlife
