Design Considerations

A Search Domain is used to quickly and easily find data. It is not used to do reporting such as 'find all orders for this particular customer created in February'.

Search Domain size
Size matters. The more attributes included in your search domain, the more expensive to create and synchronize the index. Simple attributes in your main object doesn't have a very big impact but aggregates and references do. Thus, if you define a large search domain object structure, indexing will take time. Search domain size will also have a big performance impact on generating the 'snippet' texts (the short summary text on a search hit).
Including LOB attributes
LOB columns are considerably more expensive to read than 'ordinary' columns, thus, including LOB attributes will affect indexing performance.
Keeping the Search Domain index up to date
There are two Scheduled Tasks created to keep the index up to date - Synchronize and Rebuild. Synchronization is relatively cheap since it will only process data that has been modified, added or deleted since the index was last updated. Rebuilding the index is very expensive since all the data will be indexed from scratch. You should try to design you search domain such that you don't have to Rebuild your index often.

Try to consider the following rules when designing your search domains:

If a LOB value has been changed you will have to Rebuild the index in order to include the new value.
If a detail object (arrays and aggregates) has been changed (including attributes in those objects) Synchronize will suffice.
If a referenced object has been changed you will have to Rebuild the index in order to include the new value.