Similar to Scott Boston’s suggestion, I suggest you explode the columns separately, then merge them together.
For example, for ‘Job position’:
>>> df['Job position'].apply(pd.Series).reset_index().melt(id_vars="index").dropna()[['index', 'value']].set_index('index')
value
index
0 6.0
1 2.0
2 1.0
1 6.0
And, all together:
df = pd.DataFrame({'Job position': [[6], [2, 6], [1]], 'Job type': [[1], [3, 6, 5], [9]], 'id': [3, 4, 43]})
jobs = df['Job position'].apply(pd.Series).reset_index().melt(id_vars="index").dropna()[['index', 'value']].set_index('index')
types = df['Job type'].apply(pd.Series).reset_index().melt(id_vars="index").dropna()[['index', 'value']].set_index('index')
>>> pd.merge(
pd.merge(
jobs,
types,
left_index=True,
right_index=True),
df[['id']],
left_index=True,
right_index=True).rename(columns={'value_x': 'Job positions', 'value_y': 'Job type'})
Job positions Job type id
0 6.0 1.0 3
1 2.0 3.0 4
1 2.0 6.0 4
1 2.0 5.0 4
1 6.0 3.0 4
1 6.0 6.0 4
1 6.0 5.0 4
2 1.0 9.0 43