Pandas filter: 11 ways to filter a DataFrame in Python

A pandas filter operation selects only the rows or columns that match a condition, whether that is a Boolean mask, a label or a regular expression.

This guide covers 11 practical ways to filter a DataFrame in Python. It starts with the classic beginner methods and adds the built-in filter() function, isin() and range-based filtering that other guides skip.

Every example builds on one DataFrame loaded from a CSV file, so you can follow along and swap in your own dataset.

What is a pandas filter

A pandas filter is any operation that returns a smaller DataFrame containing only the rows or columns you actually need.

You might filter by time interval in a time series, drop rows with missing data before analysis or split a dataset into subsets for testing a machine learning model. The condition can check a single column or several at once.

Pandas gives you more than one way to write that condition. Some methods read like plain Python. Others read closer to SQL. Picking the right one comes down to readability and how many conditions you’re combining.

Methods for filtering Pandas DataFrame in Python

For Filtering Pandas DataFrame let’s first create a DataFrame using a CSV file.

import pandas as pd
df = pd.read_csv('recent-grads.csv')
print(df.shape)
df.head()

Here we first imported the pandas as pd, then created the DatFrame using a CSV file, the name of the CSV file is recent-grads.csv and after that, we printed the df.shape and head of the DataFrame using df.head().

Output:

Also Read: Create a Pandas DataFrame from Lists

Methods for filtering Pandas DataFrame in Python are given below:

• Filtering DataFrame using boolean masking
• Filtering DataFrame using loc() method
• Filtering DataFrame using query() method
• Filtering DataFrame using iloc() method
• Filtering DataFrame using isnull() method
• Filtering DataFrame using the filter() method
• Filtering DataFrame using isin()
• Filtering DataFrame using string methods
• Combining conditions with AND, OR and NOT
• Filtering with nlargest() and nsmallest()
• Filtering a range of values with between()

1. Filtering DataFrame using boolean masking

We can use Boolean Mask Method for filtering the DataFrame by specifying the conditions.

Example 1:

import pandas as pd
df1 = df[ df['Major_category'] == 'Social Science' ]
print(df1.shape)
df1.head()

Here we have made a condition df[‘Major_category’] == ‘Social Science’ means anywhere the column Major_category is equal to Social Science then those rows will be returned. We saved the result in a new DataFrame df1 and then printed the size and head of the DataFrame df1.

Output:

Example 2:

import pandas as pd
filter_criteria = (df['Major_category'] == 'Social Science') & (df['ShareWomen'] >= .5)
df1m = df[filter_criteria]
print(df1m.shape)
df1m.head()

Here we have made a condition df[‘Major_category’] == ‘Social Science’ &(df[‘ShareWomen’] >= .5 means anywhere the column Major_categeory is equal to Social Science and ShareWomen is greater and equal to 0.5 i.e. the condition got true then those rows will be returned.

Output:

2. Filtering DataFrame using loc() method

We can use the loc() method where we will put certain conditions inside it for filtering the DataFrame. Using it we can return the specific columns that we want.

Example:

import pandas as pd
filter_criteria = (df['Major_category'] == 'Social Science') & (df['ShareWomen'] >= .5)
df2 = df.loc[filter_criteria,'Part_time']
df2.head()

Here we have returned the rows from column Part_time only which meets the condition (df[‘Major_category’] == ‘Social Science’) & (df[‘ShareWomen’] >= .5).

Output:

3. Filtering DataFrame using query() method

We can also use the query() method in which we can specify conditions for filtering the DataFrame.

Example:

import pandas as pd
df3 = df.query("Major_category == 'Social Science' & ShareWomen >= .5")
print(df3.shape)
df3.head()

Here we have written df3 = df.query(“Major_category == ‘Social Science’ & ShareWomen >= .5”) means anywhere the column Major_categeory is equal to Social Science and ShareWomen is greater and equal to 0.5 then those rows will be returned.

Output:

4. Filtering DataFrame using iloc() method

We can filter the DataFrame with the help of iloc() method where we need to specify the index of the rows and columns instead of the name.

Example:

import pandas as pd
df4 = df.iloc[ :10, 2:6]
print(df4.shape)
df4.head()

Here we have written df4 = df.iloc[ :10, 2:6] means we have returned the first 10 rows and columns from 2 to 6 into DataFrame df4.

Output:

5. Filtering DataFrame using isnull() method

We can filter the missing data from the DataFrame with the help of isnull() method.

Example:

import pandas as pd
df5 = df[df['Total'].isnull()]
print(df5.shape)
df5.head()

Here we returned those rows from the DataFrame where the Total column has missing values.

Output:

Once you have the rows with missing values isolated, you’ll usually want to act on them. Pandas gives you fillna() to patch the gaps and replace() to swap out chosen values, both of which pair naturally with this method.

6. Pandas filter() method: filter by column or index labels

The pandas filter() method is different from every method above. It does not check the values inside your data at all. It checks the labels, meaning column names or index values.

import pandas as pd
newdf = df.filter(items=['Major_category', 'ShareWomen'])
print(newdf.shape)
newdf.head()

This keeps only the columns named in the items list, dropping every other column from the result. Order matters here, since the output follows the order you list the names in.

The filter() method also accepts like and regex instead of items.

import pandas as pd
regexdf = df.filter(regex='Women$', axis=1)
print(regexdf.shape)
regexdf.head()

This keeps every column whose name ends in “Women”, using a regular expression instead of an exact list. The like parameter works the same way but checks for a plain substring instead of a pattern. You can only pass one of items, like or regex at a time, never more than one together.

7. Filter using isin()

The isin() method checks whether a column’s value shows up anywhere in a list you supply. It replaces a long chain of OR conditions with one clean line.

import pandas as pd
categories = ['Social Science', 'Engineering']
isin_df = df[df['Major_category'].isin(categories)]
print(isin_df.shape)
isin_df.head()

This keeps every row where Major_category matches either value in the list. Add a third or fourth category and the code barely changes, unlike stacking multiple == checks with OR.

isin() also works well with values pulled from another source, like a list built from user input or a separate query result. You can pass in a Python list, a set or even another column, and pandas checks membership the same way each time.

Negate it with the tilde operator if you need the opposite. df[~df[‘Major_category’].isin(categories)] returns every row whose category is not in the list, which is often shorter than writing out each exclusion by hand.

8. Filter using string methods

Pandas exposes string functions through the str accessor, which lets you filter text columns the same way you’d search a string in plain Python.

import pandas as pd
str_df = df[df['Major'].str.contains('Engineering', na=False)]
print(str_df.shape)
str_df.head()

The na=False argument stops the check from crashing on missing values, since contains() would otherwise return NaN for blank cells. str.startswith() and str.endswith() work the same way for matching the beginning or end of a string instead of anywhere inside it.

Case sensitivity trips up beginners here. contains(), startswith() and endswith() all check exact case by default, so “engineering” and “Engineering” won’t match unless you pass case=False.

You can also combine str methods with the tilde operator to filter out matches instead of keeping them, the same way you would with isin() or a plain condition.

9. Combining conditions with AND, OR and NOT

Real filtering rarely stops at one condition. Pandas uses &, | and ~ instead of Python’s and, or and not, because these operators work element by element across a whole column.

import pandas as pd
and_df = df[(df['ShareWomen'] >= .5) & (df['Total'] > 1000)]
or_df = df[(df['ShareWomen'] >= .5) | (df['Total'] > 1000)]
not_df = df[~(df['ShareWomen'] >= .5)]
print(and_df.shape, or_df.shape, not_df.shape)

Every individual condition needs its own set of parentheses, or Python misreads the operator precedence and throws an error. The & symbol keeps rows meeting both conditions, | keeps rows meeting either one and ~ flips the whole condition to its opposite.

10. Filter with nlargest() and nsmallest()

Sometimes you don’t have a fixed threshold. You just want the top or bottom few rows by a column’s value, and nlargest() and nsmallest() do exactly that without writing a condition at all.

import pandas as pd
top5 = df.nlargest(5, 'Total')
bottom5 = df.nsmallest(5, 'Total')
print(top5.shape, bottom5.shape)

This returns the 5 rows with the highest Total and the 5 with the lowest, sorted for you automatically. It’s a faster path than sorting the whole DataFrame with sort_values() and slicing the head afterward.

Both methods take an optional keep argument for ties. Use keep=’first’ to keep the earliest matching row, keep=’last’ for the latest, or keep=’all’ to return every row tied at the boundary value instead of cutting one off arbitrarily.

They only work on a single column at a time, so if you need the top rows by more than one criterion, sort_values() with a list of columns is still the better tool.

11. Filter a range of values with between()

The between() method checks whether a column’s value falls inside a lower and upper bound, which reads cleaner than writing two separate comparisons joined with &.

import pandas as pd
range_df = df[df['Total'].between(500, 2000)]
print(range_df.shape)
range_df.head()

By default both endpoints are included in the result. Pass inclusive=”neither” if you want a strict range that excludes 500 and 2000 themselves.

between() also works on dates once a column has been converted with pd.to_datetime(), so you can filter a date range the same clean way instead of chaining two comparisons with &.

Under the hood, between() is just shorthand for (df[‘Total’] >= 500) & (df[‘Total’] <= 2000). Reach for it whenever that shorthand reads more clearly than the full comparison.

Which pandas filter method should you use

If you end up filtering the same DataFrame more than once and need to bring the results back together afterward, combining DataFrames with pandas covers merge(), join() and concat() for exactly that step.

Key takeaways

• A pandas filter keeps only the rows or columns matching a condition
• Boolean masking and loc() cover most everyday filtering needs
• query() reads closer to SQL for complex conditions
• filter() checks labels, not values, using items, like or regex
• isin() replaces long OR chains with one list
• Use & | and ~ for AND, OR and NOT, each condition in parentheses
• nlargest() and nsmallest() skip sorting when you just need the top N
• between() is cleaner than two chained comparisons for a range

Frequently asked questions

What is pandas filter used for?

A pandas filter selects a subset of a DataFrame based on a condition, letting you isolate rows or columns for further analysis, cleaning or testing without touching the original data.

What is the difference between filter() and loc() in pandas?

filter() selects by column or index labels, using names, patterns or substrings. loc() selects by actual data values or by label-based indexing, so the two solve different problems.

How do I filter a pandas DataFrame by multiple conditions?

Wrap each condition in parentheses and join them with & for AND or | for OR, for example df[(df.a > 5) & (df.b < 10)].

Does df.filter() change the original DataFrame?

No. filter() returns a new DataFrame and leaves the original one exactly as it was, so you can chain it safely without side effects.

How do I filter rows with missing values in pandas?

Use df[df[‘column’].isnull()] to isolate rows with missing data, or df.dropna() to remove them entirely from the result.

Can I filter a pandas DataFrame using a string pattern?

Yes. Use df[df[‘column’].str.contains(‘pattern’)] for a substring match, or filter(regex=’pattern’) to match column names against a regular expression.

Conclusion

Pandas gives you more than one way to filter a DataFrame on purpose. Boolean masking and loc() handle most day-to-day work, while query(), isin() and the label-based filter() method exist for the cases where those get clunky. Pick the one that reads clearest for your condition, not the one that looks the most clever.