What algorithm can be used for packing rectangles of different sizes into the smallest rectangle possible in a fairly optimal way?

See this page on the ARC project for a survey of solutions, there is a trade-off between implementation complexity/time and optimality, but there is a wide range of algorithms to choose from.

Here’s an extract of the algorithms:

1. First-Fit Decreasing Height (FFDH) algorithm
   FFDH packs the next item R (in non-increasing height) on the first level where R fits. If no level can accommodate R, a new level is created.
   Time complexity of FFDH: O(n·log n).
   Approximation ratio: FFDH(I)<=(17/10)·OPT(I)+1; the asymptotic bound of 17/10 is tight.

2. Next-Fit Decreasing Height (NFDH) algorithm
   NFDH packs the next item R (in non-increasing height) on the current level if R fits. Otherwise, the current level is “closed” and a new level is created.
   Time complexity: O(n·log n).
   Approximation ratio: NFDH(I) <= 2·OPT(I)+1; the asymptotic bound of 2 is tight.

3. Best-Fit Decreasing Height (BFDH) algorithm
   BFDH packs the next item R (in non-increasing height) on the level, among those that can accommodate R, for which the residual horizontal space is the minimum. If no level can accommodate R, a new level is created.

4. Bottom-Left (BL) Algorithm
   BL first order items by non-increasing width. BL packs the next item as near to the bottom as it will fit and then as close to the left as it can go without overlapping with any packed item. Note that BL is not a level-oriented packing algorithm.
   Time complexity: O(n^2).
   Approximation ratio: BL(I) <= 3·OPT(I).

5. Baker’s Up-Down (UD) algorithm
   UD uses a combination of BL and a generalization of NFDH. The width of the strip and the items are normalized so that the strip is of unit width. UD orders the items in non-increasing width and then divides the items into five groups, each with width in the range (1/2, 1], (1/3,1/2], (1/4,1/3], (1/5,1/4], (0,1/5]. The strip is also divided into five regions R1, ··· , R5. Basically, some items of width in the range (1/i+1, 1/i], for 1 <= i <= 4, are packed to region Ri by BL. Since BL leaves a space of increasing width from top to bottom at the right side of the strip, UD takes this advantage by first packing the item to Rj for j = 1, ··· , 4 (in order) from top to bottom. If there is no such space, the item is packed to Ri by BL. Finally, items of size at most 1/5 are packed to the spaces in R1, ··· , R4 by the (generalized) NFDH algorithm. Again if there is no space in these regions, the item is packed to R5 using NFDH.
   Approximation ratio: UD(I) <= (5/4) · OPT(I)+(53/8)H, where H is the maximum height of the items; the asymptotic bound of 5/4 is tight.

6. Reverse-fit (RF) algorithm
   RF also normalizes the width of the strip and the items so that the strip is of unit width. RF first stacks all items of width greater than 1/2. Remaining items are sorted in non-increasing height and will be packed above the height H0 reached by those greater than 1/2. Then RF repeats the following process. Roughly speaking, RF packs items from left to right with their bottom along the line of height H0 until there is no more room. Then packs items from right to left and from top to bottom (called reverse-level) until the total width is at least 1/2. Then the reverse-level is dropped down until (at least) one of them touches some item below. The drop down is somehow repeated.
   Approximation ratio: RF(I) <= 2·OPT(I).

7. Steinberg’s algorithm
   Steinberg’s algorithm, denoted as M in the paper, estimates an upper bound of the height H required to pack all the items such that it is proved that the input items can be packed into a rectangle of width W and height H. They then define seven procedures (with seven conditions), each to divide a problem into two smaller ones and solve them recursively. It has been showed that any tractable problem satisfies one of the seven conditions.
   Approximation ratio: M(I) <= 2·OPT(I).

8. Split-Fit algorithm (SF)
   SF divides items into two groups, L1 with width greater than 1/2 and L2 at most 1/2. All items of L1 are first packed by FFDH. Then they are arranged so that all items with width more than 2/3 are below those with width at most 2/3. This creates a rectangle R of space with width 1/3. Remaining items in L2 are then packed to R and the space above those packed with L1 using FFDH. The levels created in R are considered to be below those created above the packing of L1.
   Approximation ratio: SF(I) <= (3/2) ·OPT(I) + 2; the asymptotic bound of 3/2 is tight.

9. Sleator’s algorithm
   Sleater’s algorithm consists of four steps:

   1. All items of width greater than 1/2 are packed on top of one another in the bottom of the strip. Suppose h0 is the height of the resulting packing All subsequent packing will occur above h0.

   2. Remaining items are ordered by non-increasing height. A level of items are packed (in non-increasing height order) from left to right along the line of height h0.

   3. A vertical line is then drawn in the middle to cut the strip into two equal halves (note this line may cut an item that is packed partially in the right half). Draw two horizontal line segments of length one half, one across the left half (called the left baseline) and one across the right half (called the right baseline) as low as possible such that the two lines do not cross any item.

   4. Choose the left or right baseline which is of a lower height and pack a level of items into the corresponding half of the strip until the next item is too wide.

   A new baseline is formed and Step (4) is repeated on the lower baseline until all items are packed.
   Time complexity: O(n ·log n).
   The approximation ratio of Sleator’s algorithm is 2.5 which is tight.