PyCon Israel 2024 - Conference
Schedule
Discover fascinating world of real-time image and video processing using Python. Learn how we protect women from violent ex-partners by leveraging Python libraries, algorithms, and deep learning models in our Michal Sela Hackathon project.
Whether you're a junior software engineer exploring the realm of image processing or a seasoned developer seeking inspiration, join me on a journey through the fascinating world of real-time image and video processing using Python.
In this talk, I'll share insights from a recent Hackathon project where we tackled the challenge of protecting women from their ex-partner following violent relationship.
I will demonstrate (with code snippets and working examples) how leveraging Python's robust open-source libraries and frameworks, latest algorithms, and deep learning models we have crafted, in a short time, a working solution that pushes the boundaries of traditional image processing techniques.
Furthermore, we'll take you through our Hackathon journey, sharing the highs and lows, the triumphs and setbacks, and the invaluable lessons learned along the way.
We call software and algorithms engineers to join forces and contribute to safe-tech projects like ours. Together let’s make a life-saving impact through innovation.
Every developer uses open-source packages and models. Only a fraction of us validate their security. This session will cover the supply chain security issues that Python developers face, show attacks, and recommend how to avoid them.
PyPi has 1.5 billion packages downloaded daily. This huge number is the perfect opportunity to disguise a malicious needle in the package haystack. Due to its popularity among Python developers, PyPi is also extremely popular among attackers. Same for HuggingFace, which gains its popularity with skyrocketed usage of models. Attackers may target masses using techniques like typosquatting or perform targeted campaigns against maintainers of the top projects and even the whole developer communities. One of the crucial skills that a Python developer must have nowadays is the responsible use of open-source dependencies. The talk will focus on the issues that may happen to the developers and the way of avoiding them.
Python’s nature often invites slip-ups: We’ll explore tools that prevent (some) bugs and improve code quality, discuss strategies for adding them to existing projects, and share tips for keeping your repo tidy and your fellow developers happy.
Ew, what’s that smell? Is it a non-typed, 15-argument, Python2-styled function, with a mutable-default argument and unreachable code again?
Your code is your kitchen: bugs hide best when it’s dirty🪳
Luckily, there are awesome tools to help reduce code smells, improving the quality, safety and consistency of your beloved Python code! 🧹
In the talk, we’ll explore a range of these tools, see how to fit them to your project’s needs, integrate them into IDEs and CI pipelines, choose the right rules to enforce, and apply them to your codebase without annoying your coworkers too much
Discover ‘One Million Hands,’ an advocacy platform addressing Israel’s advocacy needs. Learn how we evolved from simple Python scripts to a robust platform using data science, machine learning, and Python to empower effective social media advocacy.
“In early October, we launched ‘One Million Hands’ to address the need for effective advocacy in Israel. Starting with simple Google Sheets and Python scripts, our project has evolved into a robust platform empowering individuals to impact social media conversations through optimal feedback on targeted content. Leveraging autonomous tools and a comprehensive data science pipeline, we enrich content behavior analysis by integrating data from multiple sources and applying machine learning models to tag content violations.
As a voluntary organization, we face challenges in long-term development, managing personnel, and creating robust, organized code. Our platform gathers crucial information from social networks, efficiently funneling data through our pipeline and data science services. Python has been instrumental in managing our systems, offering flexibility in volunteer contributions, knowledge preservation, and interfacing with various programming and cloud tools.
In the lecture, we will present the technical journey of ‘One Million Hands,’ focusing on Python’s role in data analysis, machine learning, and API development. Gain insights into practical applications of Python in a real-world project, the challenges we faced, and our solutions. A live demonstration of our platform will highlight how technology drives impactful advocacy.”
Introducing secimport, an eBPF-based sandbox toolkit for Python, enforcing specific syscalls per module. It traces, profiles, and enforces security policies, offering granular control and reducing attack surfaces, with negligible performance impact.
Introducing secimport, an eBPF-based sandbox toolkit designed to secure your Python applications by enforcing specific syscalls per module. Think of it as seccomp-bpf for individual modules in your code.
Traditional tools like seccomp and AppArmor enforce syscalls for the entire process, which isn't enough for Python's complex attack surface. Secimport traces the syscalls each module uses, creating a security profile that’s compiled into a lightweight eBPF script. This script enforces the profile during runtime, ensuring your application behaves as expected.
The need for secimport arises from the limitations of current security tools that apply blanket syscall restrictions, which fail to address the nuanced and varied needs of individual Python modules. Secimport solves this problem by allowing granular control at the module level, effectively reducing the attack surface and mitigating risks associated with supply chain attacks and insecure code execution. This level of fine-tuned control is only feasible with eBPF, a powerful technology that allows high-performance, real-time instrumentation and monitoring at the kernel level, enabling us to create robust, low-overhead security profiles for Python applications for the first time.
Secimport is perfect for preventing code execution attacks and securing your supply chain. It minimizes the risk of vulnerabilities by confining each module to only the syscalls it needs. Whether you’re dealing with AI models from insecure sources or just want an extra layer of defense, secimport has you covered with negligible performance impact.
This talk will dive into how secimport works, from tracing to policy enforcement, with practical examples and benchmarks. You’ll learn how to set it up in various environments, including Docker and macOS, and see real-world use cases that highlight its effectiveness.
Text clustering is a fundamental process in NLP, but what do you do when your clusters just aren’t right? I will share my journey where I ended up combining sklearn and langchain to reduce duplication and "Misc" clusters.
Text clustering can be used to organize text, analyze data, help extract topics or segment customers by interests. In addition, we are blessed with many cheap and high-quality text embedding APIs that should improve clustering. But what do you do when your clusters just aren’t right? Topics are duplicated, clustering is too sensitive to vocabulary, and there’s always that one giant “Misc” cluster.
As a newcomer to clustering, I gravitated towards KMeans since it seems to be the default clustering algorithm. I will share the pitfalls I encountered, the clustering algorithms I explored and how I incorporated LLMs into the process to achieve far better results.
Discover how to enhance your Python apps with plugin architectures using importlib.
In today's rapidly evolving software landscape, applications often need to be flexible and extensible to meet diverse user requirements. A plugin architecture allows users to enhance the capabilities of your application dynamically. This talk is aimed at developers who want to learn how to design and implement such an architecture in their Python applications.
We will begin by discussing the scenarios where a plugin architecture is beneficial (the why). Next, we will dive into the technical details, presenting various strategies for implementing a plugin system (the how). These strategies include:
- Simple dynamic imports for lightweight extensions.
- Using entry points for more structured and discoverable plugins.
- A registration system that allows plugins to declare capabilities and dependencies.
- Improving performance - loading on the fly
We'll shortly discuss the challenges you might face, such as maintaining code quality, ensuring security, and managing dependencies.
To make these concepts concrete, we'll walk through a real-life example where I integrated a plugin system into a Jinja-based application, enabling users to register custom Jinja filters. This example will demonstrate how to leverage importlib to dynamically load and register user-defined plugins safely and efficiently.
Target Audience:
This talk is intended for Python developers of all levels who are interested in enhancing their applications with dynamic, user-defined extensions. Whether you're working on a personal project or a large-scale application in an organizational setting, you'll find valuable insights and practical techniques that you can apply immediately.
Mobileye accelerates ML development with multi-modal datasets using Python, Parquets, and Daft. We will cover dataset formats, Daft’s capabilities, its usage examples, and its integration into Mobileye's cloud-native architecture.
Large-scale multi-modal datasets play a pivotal role in the development of computer vision solutions based on deep learning algorithms. Historically, different data formats have been used at various stages of the ML development life cycle to optimize specific tasks. During training, a sequential pass over the entire dataset is essential, while validation involves map-reduce operations.
In this talk, we’ll delve into how Mobileye leverages Python, Parquets, and Daft to streamline the AI development life cycle. We’ll explore the following key aspects: - Dataset Formats and Reading Options - We’ll discuss various options for representing multi-modal datasets, and how choosing the right format impacts training and validation efficiency. - What Is Daft - Daft is a high-performance Python query engine designed for handling complex, multimodal data types. Its Rust core engine executes operations lazily via Daft’s Expressions API. Additionally, Daft seamlessly integrates with essential Python libraries like PyTorch and offers efficient cloud storage integration. - Examples of Daft Usage - Through practical examples, we’ll demonstrate how Daft simplifies working with multi-modal datasets. From loading data to preprocessing. - Daft in a Cloud-Native Architecture - Learn how Daft seamlessly integrates into Mobileye’s cloud infrastructure. We’ll explore its role in enabling multi-tenant access to datasets, ensuring fast data retrieval, and maintaining a single source of truth.
Throughtout the talk We’ll discuss real-world scenarios showing how Daft accelerates the AI development lifecycle. Attendees will gain insights into best practices for leveraging Daft effectively.
Delve into the inner workings of Python's import system. Learn how standard imports are implemented, and how to extend the import statement to provide new functionality.
The Python import system is more dynamic and extensible than most people realize. While most users primarily use it to import built-in and installed packages, there are additional options available, such as importing from zip packages and C modules. Moreover, Python allows you to hook into the import machinery to write your own import procedures. In this talk, we will explore how the import system works behind the scenes. We will cover finders, loaders, and sys.meta_path. How the default file importer works, explaining sys.path, PYTHONPATH and how relative imports work. We will see how these are implemented in Python with code snippets. We will see 2 example usages of import hooks, one practical, the other, extremely ridiculous.
We built a python application that ingests 100k events per second. This lecture will share our lessons learned using different dbs for this job.
Our python based system needs to handle more than 100k events per second. During the last two years we have tried, failed and evolved using multiple databases such as nosql, timeseries db and OLAP columnar dbs. This lecture will share the lessons learned for running such a system in a production environment
Join me as I present my experience of teaching Python to 11-year-old girls through AliceCode’s program. Discover which concepts were challenging for them, examples of the coding exercises used, samples of their projects, and additional insights.
How do you teach 11-year-old girls Python? AliceCode is a non-profit organization whose goal is to train and empower young girls, from the fourth grade and up, in the software field. Through AliceCode, I had the privilege of teaching Python programming to young girls from various parts of the country via Zoom in their first steps into the world of coding.
During the session I will speak in-depth about the insights I gained in the process of teaching Python to young girls, including an overview of the program’s syllabus, which concepts were the hardest for them to understand, examples of the coding exercises and questions used during lessons, and samples of the students’ completed projects including code snippets.
I will also briefly share my firsthand observations of the outcomes and impact of early exposure to coding, and shortly discuss how I believe parents can motivate their children, specifically girls, to pursue and stay in computer science.
Navigating Python dependencies can be a nightmare, especially with large frameworks like Apache Airflow. Join Roy as he shares the Wix way of keeping dependencies updated and stable,ensuring a true sleeping beauty in the world of software development
Any experienced Python developer knows that Navigating the complex landscape of Python dependencies can be akin to a nightmare, especially when dealing with large frameworks like Apache Airflow which has over 700 Python dependencies. In This talk Roy will share the Wix way of keeping dependencies updated and most importantly stable!
Join us for a deep dive into the world of Python dependency management with a focus on stability and sanity. In this talk, Roy will uncover the challenges faced by developers dealing with an extensive array of Python dependencies, particularly within frameworks like Apache Airflow and proprietary in-house libraries. Drawing from his experience at Wix, where the Python ecosystem is rich and diverse, with more than 1,000 dependencies that serve ~200 Data Engineers, we share insights into our battle-tested methods for keeping dependencies up-to-date and rock-solid. In this talk the audience will learn practical solutions and best practices for keeping their environment stable. We will discuss several ways to effectively manage and handle custom dependencies in your python platform.
Whether you're an experienced Python developer or just starting out, join me to unlock the secrets to a tranquil dependency experience – a true sleeping beauty in the world of software development.
Developers who are not familiar with databases often dread them and treat them like blackboxes, but fear no more! In this talk I present advanced indexing technics to make your database faster and more efficient.
Indexes are extremely powerful and ORMs like Django and SQLAlchemy provide many ways of harnessing their powers to make queries faster and the database more efficient. In this talk I reveal the secrets of DBAs with some advanced indexing techniques such as partial, function based and inclusive B-Tree indexes, and who knows, maybe even some index types you never heard of before!
Discover Python's Abstract Syntax Tree (AST), and learn how this fundamental data structure simplifies code analysis and metaprogramming through real-life examples.
At first glance static analysis of code may seem daunting. Recognizing patterns in code as text is second nature to us, but attempting this programmatically can be highly cumbersome. Yet, this incredibly useful capability is a cornerstone of many Python tools we rely on to enhance our code and preemptively catch potential bugs. This is where the Abstract Syntax Tree (AST) comes in - a powerful data structure that Python uses to represent code, making many challenges of code analysis and metaprogramming much simpler.
In this talk, we will delve deep into Python's AST, offering you a fresh perspective on the syntax of Python and programming languages in general. Through practical examples, you'll learn how to implement static analysis and metaprogramming using AST, empowering you to harness this powerful tool yourself.
While profiling a slow process I stumbled upon a surprising way to reduce our memory consumption. This talk will present some useful profiling tools, and an important thing to know when using AbstractBaseClass extensively.
In this session, we will dive into the realm of Python optimization, as we cover some essential profiling tools designed to identify and resolve performance bottlenecks in your code. We'll navigate through practical examples, showcasing how these tools can provide invaluable insights into your application's memory and CPU usage patterns.
Furthermore, we'll delve into some nuances of AbstractBaseClass usage, and its implications on speed and memory management in Python applications.
Whether you're a seasoned developer or just starting your journey with Python, this session offers some practical strategies to optimize Python programs effectively.
Reviewing code repetitively and documenting API endpoints can be tedious. In this talk, we’ll share how we wrote Python LLMs with Github actions and a few python SDKs for best practice advice on pull requests and automatic API documentation.
Being a software engineer is awesome. I love waking up in the morning and diving into code, planning new system components, or collaborating with my team to release great versions for our users. But doing the grunt work? I’ll pass. While I enjoy reviewing other developers’ architecture and learning from their approaches, being the team’s bottleneck, only to give the same repetitive “best practices” comments? No thanks. Speed coding new endpoints for great frontend capabilities? LOVE. Adding API endpoint documentation? Please, spare me the boredom. We keep talking about “how amazing AI is going to transform our products,” but what really amazed me was the morning I realized I could turn Gemini into a sidekick that handles my dirty work.
In this talk, we’ll share how we used Gemini to offload tedious tasks and let us focus on the enjoyable parts of development. We’ll walk you through how we wrote a Python LLM that instantly provides best practice advice on GitHub pull requests and demonstrate how another LLM handles API documentation for us. We’ll also present other practical uses that might benefit your team. Additionally, we’ll discuss important considerations before starting, how we improved our processes, and what made the journey fun. This talk is perfect for anyone looking to take their first steps in writing LLMs, those who constantly seek to optimize their daily lives, and of course – Python enthusiasts.
Ever wondered why your I/O loop gets stuck in production? We'll explore Python's I/O loop with asyncio, share a real-life bug, explain async and await, and introduce our monitoring solution to prevent and resolve issues. Join us to learn more.
Ever wondered why your I/O loop gets stuck in production? In this talk we'll dive into the world of Python's asynchronous programming and I/O loop. We'll start with an introduction to event loops, using the built-in asyncio library. Then, we'll cover a real-life bug where a stuck I/O loop caused crashes and general slowness in our production environment. Afterwards We'll break down the mechanics of the I/O loop, explain terms like async, await, and coroutines. Finally, we'll talk about our solution for monitoring I/O loops in production, ensuring you can prevent and resolve issues. Join us to deepen your understanding of the I/O loop and learn new techniques for keeping your systems running smoothly.
We will take a small demo library which uses numpy, and improve its performance 100x using the py-spy sampling profiler and Rust
Python is awesome, but can be super slow
Even when using packages like numpy, we can still be limited by Python's general slowness.
In this talk, I'll walk you through taking a demo small library which uses numpy, and building a native extension to it with Rust, that improves performance by 100x, all while keeping the original API the same.
In this lecture, we will explore the evolution of Python monitoring over the years, covering tools and techniques from sys.monitoring to import hooks, highlighting advancements and best practices in keeping your Python code in check.
Join us and time-travel across the evolution of Python monitoring mechanisms. We'll delve into history from dedicated tools like sys.monitoring to more advanced techniques such as ceval and import hooks. This session will provide a comprehensive overview of how monitoring practices have developed over the years, offering insights into the best practices for maintaining and debugging your Python code and the pros and cons of each approach. Whether you're a seasoned developer or new to Python, you'll gain valuable knowledge on how to keep your code running smoothly and efficiently without hurting performance or your dev velocity with tedious maintenance.
A Python unit-test framework - for making sure that Agents do as they're expected.
LLM Agents are powerful constructs - capable of asking questions, using tools, reasoning iteratively while fixing their mistakes or assumptions. When creating such agents for a specific purpose, we want to make sure that purpose is achieved. However, because these beings exist in a conversational, non-deterministic environment, doing proper unit tests becomes tricky. In this talk I will present a method, as well as a Python framework, for tackling this problem.
Airflow is great, but it's notoriously hard to avoid duplicate code. We'll show you how we got into technical debt, and how we got out of it. We'll share our powerful and reusable class library, and the stories behind its development.
The challenge of writing reusable Airflow code, and the nightmare of technical debt. How we built a powerful and easy to use class library for our Airflow DAGs. What to do -- and what not to do -- when refactoring. Development tradeoffs, and managing technical debt.
Pydantic is the most widely used data validation library for Python, and JSON is the most common format for exchanging data. Take those together, and you too will never worry about sending, serializing, or creating valid data structures again.
As someone who had the pleasure of trying and creating clients for an OpenAPI, I know that one of the most annoying things to do is to work with a pythonic OpenAPI client that is too generic, only giving you the bare minimum when it comes to sending requests, not having any validation when it comes to sending data to the server, and leaving it up to you to hardcode everything.
If you ever needed to maintain complex data models in your code for any reason, you know it’s a hard-to-manage task, from forgetting what models were added and where, to what dependencies each model has, having duplicated models, to just having a messy codebase that is impossible to navigate.
Modern problems require modern solutions, so why not use the most popular data validators? You might already be familiar with JSON Schema from interacting with it in Postman, Swagger, Redoc, or any other API documentation that exists. It’s a way to define a template for how a JSON object should look. Pydantic, like its name suggests, is a pedantic library that lets you create data models but does not let you make any typing mistakes. Combining the two, we can create a generator, a validator and a serializer all in one.
In this talk, we will discuss: 1. JSON Schema and what we can do with it. 2. Why trying to structure complex data models in code is generally a bad idea. 3. How to generate an OpenAPI client using Python.
Explore the transition from developer to security researcher, addressing log safety in applications. Learn common mistakes, practical Python solutions, and empower developers to protect against data exposure.
In my seven years as a software developer, I've primarily worked in teams composed solely of developers. However, my recent transition to a team of security researchers has opened my eyes to a crucial aspect that often goes unnoticed: log safety in applications.
My exposure to the application security ecosystem and real-life security breach analysis has opened my eyes to recognize code security issues, including the prevalence of sensitive information, tokens, passwords, and payment details, in plaintext logs. This may lead to severe data breaches, financial losses, and all kinds of catastrophes.
This talk will dive into the fatal mistakes developers often make that can result in the disclosure of sensitive information in logs. We'll explore the types of sensitive data in logs.
I'll share my personal experiences as a developer on a security research team and shed light on the often-overlooked consequences of insecure logging practices. We'll discuss practical patterns to safeguard sensitive information in Python applications, including identifying and redacting sensitive data before it reaches log files, and implementing secure logging practices.
By the end of this talk, developers will be equipped with the knowledge and tools to protect sensitive data from accidental disclosure and safeguard their applications from the perils of sensitive data exposure. Embrace the journey towards log safety and ensure your code remains secure and confidential.
Cryptography is a vital part of application security, but the world of cryptography is dark and full of errors. We will talk about deprecated algorithms, poor key management, misuse of libraries, and best practices for secure implementation.
This session will discuss common mistakes developers make when implementing cryptography in Python. There are many. Through real-world issues and cases (yes, also embarrassing ones), we'll illustrate the consequences of these bad practices and provide practical recommendations for secure cryptographic implementation. A must-have for every developer that is working on a back-end,
Dive into our Python-powered adventure of transforming a hodgepodge of bash scripts into a sleek, unified Python package for our entire CI toolkit. We'll share how our small team, armed with Python and determination, filled the DevOps void.
In this talk, we will be exploring how we embraced the fact that Valerann is a Python company and how we implemented it in everything that we do - even in our CI tools.
We use microservice architecture, manage 10+ AWS accounts, do weekly production releases, use various monitoring tools and actively develop the product.
Two years ago, our development tools where scattered amongst various off-the-shelf products and custom-built tooling in a variety of languages.
In the time since, we have fully switched over our development tools to be a singular Python package. This allows us to define containers, services, monitoring, alerting, testing and much more using the language all our developers are most comfortable with. Gone are the days of writing bash scripts, yaml definitions and json blobs!
We'll talk about what we had before, how is was implemented, how we did the same in Python:
- New repo scaffolding
- Monitoring and alerting (Grafana)
- Infrastructure (Terraform)
- Building containers
- AWS deployment
Tests can be great, but sometimes they're really not. Getting it right can be tricky, and even when there are problems - it can be hard to know why. We'll see how to avoid some common expensive mistakes, and how this can look in the Python ecosystem.
Almost every developer who’s worked with tests has encountered a test suite that caused a lot of pain.
Some of them just don’t protect us when we need them, some are flaky, some keep breaking because of unrelated changes, some take hours to debug whenever they fail.
And while every company is different, there are definitely patterns. Many problems are the result of some common pitfalls that trap many teams. These pitfalls might be common, but they're not easy to spot - small differences in test implementation can dramatically affect the result, and I’ve seen all of them happen in strong, capable, experienced teams. Most of these I fell into myself at least once.
In this session, we'll take a look at a selection of problematic testing choices, with examples that show these in the context of common Python frameworks and libraries. We'll discuss how to identify them, what problems they might cause and what alternatives we have so we can save ourselves the pain.
