Agents cannot break the laws of physics. Babies cannot solve maths problems. A person with a broken arm cannot lift heavy objects. Someone living in poverty cannot buy a house. There are many things that agents cannot do, and it seems that there are different types of limitations on an agent’s abilities depending on the factors involved (e.g. physical, mental, social or normative). Some limits apply to humans in general and some only to individuals. Some limits can be overcome, while others cannot. Some are viewed as deficiencies, while others are not. This raises various questions, such as: What types of factors limit an agent’s abilities? How do these factors limit our abilities, e.g. are they external or internal to these abilities? How do normative factors come into play? Do they constrain our abilities, or affect how they are evaluated? The limits of abilities and the corresponding questions and challenges have not yet been systematised. In my talk, I will make a first attempt to do so.