Innovation: Array quirks and Generators

ListsSpeaking a little bit about innovation, we had a great discussion about Class Signatures also here in this blog as well as in a very hot thread in the .language forum, and finally I’ve submitted also a QC Report – vote for it btw.

But today I would ask for your opinion about another feature request, which, in fact, will fill a gap (or rather an oddity?) in the Delphi compiler. But the main point is how to implement it…

Now we can have things like this:


type
   TShopItem = record
     Name : string;
     Price : currency;
   end;

const
   Days : array[0..6] of string = ('Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat') ;

   CursorMode : array[boolean] of TCursor = (crHourGlass, crSQLWait) ;

   Items : array[1..3] of TShopItem =
   (
     (Name : 'Clock'; Price : 20.99),
     (Name : 'Pencil'; Price : 15.75),
     (Name : 'Board'; Price : 42.96)
   ) ;

So we can initialize a const. So far so good. (Example taken from Zarko’s delphi.about.com).

But we can even do like this:


var
   Days : array[0..6] of string = ('Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat') ;

We can also initialize a variable. (But of course this doesn’t work for local variables – vote for the suggestion) But then why we cannot assign one in the same way?

Something like: (hold on guys, from now on the feature requests are coming)


var
   Days : array[0..6] of string = ('Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat') ;

begin
  if CheckForLargeNames then
    Days:= ('Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday') ;
end;

The above, as you know, won’t compile. 😦

Also, in the case of dynamic arrays the assignment would set also the size of the array:


var
   Days : array of string = ('Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat') ;

begin
  if CheckForLargeNames then
    Days:= ('Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday') ;
end;

Hmmm… but I want that the days of the week to start from 1… So what if I can write something like…


var
   Days : array [1..] of string = ('Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat') ;

begin
  if CheckForLargeNames then
    Days:= ('Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday') ;
end;

…and let the compiler to decide till where to expand the array.

Hmmm… (again) …but what if my list will be generated upon request, based on a function (using a hidden enumerator)? I need many times to write something clear and concise things like this:


var
   a : array of double;

begin
  a := (for i in [1..4]); //gives (1,2,3,4)
  a := (for i in [1..4] do i/2); //gives (0.5, 1, 1.5, 2)
  a := (for i in [1..6] do i/2 if i mod 2 = 0); //gives (1,2,3)
  a := (for i in [1..10] step 2); //gives (1,3,5,7,9)
end;

The general syntax would be

a:=”(” for <counter> in <range> [step <step>] [do <counter processing>] [if <filtering condition>] “)”

Or perhaps if is better to be replaced with where?

As an aside, yes, sons, perhaps is time to implement the Step clause in For. Just a humble hint. We are in 2009, you know.

Of course this should support anonymous lists:


a:=(for i in [(for k in [1..3] do k*2)] do i+1); //gives [3,5,7]

//which will allow us to pass as an argument like in...
procedure MyFunc((for i in [1..10)); //Aggregate functions like Sum, Avg etc. are fist candidates here.

Also, as you can comprehend ( 😉 ) this allows expressing infinite lists because the n-th element is generated on demand using the hidden enumerator inside.

And before you bite, yes, here are two different requests: one talks about a static assignment to an array in the program body and one talks about a dynamic and and concise way to express a generator – a (perhaps infinite) list implemented using an enumerator.

And before you bite, yes, the latter “resembles a little” Python’s list comprehensions. A simple tutorial here.

And before you bite, yes, the latter “resembles a little” an old QC report on this. 😉 Vote for it.

Comments? Thoughts? Suggestions?

15 thoughts on “Innovation: Array quirks and Generators

  1. Regarding your first suggestion, I’d prefer the pseudo-constructor of dynamic array types to be extended to accept array constants. So, currently, you can do this:

    var
    Bytes: TBytes;
    begin
    Bytes := TBytes.Create(1, 2, 3);

    What I’d like, then, is if you could also do this:

    const
    SomeBytes: array[0..2] of Byte = (1, 2, 3);
    DifferentBytes: array[0..2] of Byte = (4, 5, 6);

    var
    Bytes: TBytes;
    begin
    if SomeCondition then
    Bytes := TBytes.Create(SomeBytes)
    else
    Bytes := TBytes.Create(DifferentBytes);

  2. I like the idea of being able to assign to an array variable in code, but I can understand a) why you can’t and b) why it is likely to be impractical in practice.

    When assigning to a variable in code you are removed from the declaration of the array itself so you don’t know how many elements are actually required (in the case of fixed dimension arrays). It’s one of those things that looks convenient when you sit down with a simple example but I think would be impractical in actual use.

    Dynamic arrays are a different matter and could be accommodated more practically, but that then raises the issue of treating different types of arrays very differently. Then again, dynamic arrays are already very different from static arrays – it’s only that their syntax is so similar that it’s easy to forget the important distinctions that exist between them.

    • Think at the analogy between fixed-size strings (ie. string[200]) and the variable length strings. Then your points still apply? (Ie. a fixed-size string should be initialized only at the declaration place), assigning strings is impractical for big sizes etc.
      I do think that nowadays any feature can be mis-used. It is just to see what is the advantage which brings, and if the advantage is greater than the headache then perhaps we must implement it. Also, a sign that this is useful is that other languages already have it: Python (link in post), Haskell and of course .NET’s Linq is in fact a form of it (ie. a way of generating dynamic lists based on a condition).

  3. I can pretty easily see a number of problems with your ideas and why the compiler does not allow it.

    The biggest problem is that the ( symbol has no special meaning in the context of an assignment in the same way it does that of a Const or Var assignment.

    Try this instead :

    Type
    TDays = Array[0..6] Of String;

    const
    _Days : TDays = (‘Sun’, ‘Mon’, ‘Tue’, ‘Wed’, ‘Thu’, ‘Fri’, ‘Sat’);
    _LongDays : TDays = (‘Sunday’, ‘Monday’, ‘Tuesday’, ‘Wednesday’, ‘Thursday’, ‘Friday’, ‘Saturday’);

    begin
    Days := _Days;
    if CheckForLargeNames then
    Days:= _longDays;

    AS for the rest. All I can say is YIKES. Create a type, write a function and have it return an array of that type. Something like:

    Type TArr=Arrray Of Integer;

    Function ForIn(Start,Finish : Integer) : TArr;

    Var
    Loop : Integer;

    Begin
    SetLength(Result,0);
    For Loop := Start To Finish Do
    Begin
    SetLength(Result,Length(Result)-1);
    Result[Length(Result)-1] := Loop;
    End;
    End;

    Nice and simple. Make more complicated as you need.

    Oh, and the whole dynamically sizing array is great for 2 or 3 items. Hit a few hundred and performance starts to seriously take a hit. Interesting idea, simple example – hideous result in scale. If you were to write a routine, you would want to have some clue how large the result was.

    So ya, interesting ideas – however all are already easily solvable already, some better solved with code as it allows you more control over the end result and how you get there.

    • “The biggest problem is that the ( symbol has no special meaning in the context of an assignment “
      This is a grammar / parser way of behaving not a compiler issue. And it can be changed. Anyway this isn’t the gist of the issue.
      Do you think that writing an entire function (like you did) for each case is “nice and simple” compared with (in your case) a:=(for i in [1..10]) ? Also in your function ForIn it doesn’t even show the rule. One has to look to the actual code to see what’s happening inside. So you must write a sea of functions called GetImpareNumbers, GetLogNumbers, GetSomeStrings etc. etc. to cover everything? And how will you nest these functions? All the IT crowd is going today to the lambdas and functional programming and should we stay to plain old procedure based style? Sorry but I don’t think so.
      “Oh, and the whole dynamically sizing array is great for 2 or 3 items. Hit a few hundred and performance starts to seriously take a hit.”
      1. The proposed syntax improves this. Actually a for cycle (your solution) is way slower than my syntax. Do a benchmark of 10000ths assignments compared with a for cycle on the same structure to see.
      2. So you propose to not use (so do not assign values to) arrays, strings and TStringLists bigger than 2 or 3 items because of “performance issues”?
      3. A routine is necessarily scanning the entire array from Start to Finish as you demonstrated in your example. However a generator (because it uses internally an Enumerator) it will work in the worst case (if techniques like caching/memoization, direct access aren’t used) till the requested ‘n’ element. So it is faster than your approach. And it can be considerably faster.
      4. Having a generator with an enumerator inside can also have the advantage to reduce a lot the memory used. It can use different amounts of memory (eg. the cached elements – if applicable etc.) whereas an array initialized in a For cycle needs to ‘have room’ for all elements.

  4. May be there is a way using generics:

    type
    TArray = class
    type TDynArray = array of T;
    class function Create(const Items: array of T): TDynArray;
    class procedure Init(var Dest: array of T; const Src: array of T);
    end;

    from your code:

    Days:= (‘Sunday’, ‘Monday’, ‘Tuesday’, ‘Wednesday’, ‘Thursday’, ‘Friday’, ‘Saturday’) ;

    will become (if Days is a dynamic array):

    Days:= TArray.Create([‘Sunday’, ‘Monday’, ‘Tuesday’, ‘Wednesday’, ‘Thursday’, ‘Friday’, ‘Saturday’]) ;

    or (if Days is a static array):
    TArray.Init(Days, [‘Sunday’, ‘Monday’, ‘Tuesday’, ‘Wednesday’, ‘Thursday’, ‘Friday’, ‘Saturday’]) ;

    • Some text was understood as html tags…
      I hope the < and > are now displayed (I wrote &lt; and &gt;)

      May be there is a way using generics:

      type
        TArray<T> = class
          type TDynArray = array of T;
          class function Create(const Items: array of T): TDynArray;
          class procedure Init(var Dest: array of T; const Src: array of T);
        end;
      

      from your code:

      Days:= (’Sunday’, ‘Monday’, ‘Tuesday’, ‘Wednesday’, ‘Thursday’, ‘Friday’, ‘Saturday’) ;

      will become (if Days is a dynamic array):

      Days:= TArray<String>.Create([‘Sunday’, ‘Monday’, ‘Tuesday’, ‘Wednesday’, ‘Thursday’, ‘Friday’, ‘Saturday’]) ;

      or (if Days is a static array):
      TArray<String>.Init(Days, [‘Sunday’, ‘Monday’, ‘Tuesday’, ‘Wednesday’, ‘Thursday’, ‘Friday’, ‘Saturday’]) ;

    • Jacinto, you know that Delphi 2010 already allows this without the help of generics.

      type
        TStringDynArray = array of string;
      var
        Arr: TStringDynArray;
      begin
        Arr := TStringDynArray.Create('a1', 'a2', 'a3', 'a4');
        ...
      end;
      
      
      • Yep. This is the idea. And this can be enhanced to have a more clearer syntax. But the main point is the other one: Having a clear syntax to generate the a list (array etc.). But for these we need lambdas.

        • I remember Barry Kelly talking about the problems to implement lambdas:
          The parser may need to know some types before they are available the way Delphi works today…
          May be with the new way they are working in, the problem can be solved.

          Even if it cant be solved, y the compiler can cast an expression of type T1 to a function reference of type T2 inserting the cast from T1 to T2 in the code, the next could work:

          type 
            TArray<T> = class ...
              type TDynArray = array of T;
              type TFunction = reference to function: T;
              class function CreateFor(var i: Integer; iMin, iMax: Integer;
                Fn: TFunction): TDynArray
          ...
          procedure DoSomeThing;
          var
            i: Integer;
            Arr: array of Double;
          begin
            Arr := TArray<Double>.CreateFor(i, 1, 10, i * i);
          

          or, if the dynamic arrays are nor compatible, it could work using:
          Arr: TArray<Double>.TDynArray;

          • Yep, something like this. Well even if we can improve your syntax, the biggest problem is the lambdas as you observed. Using now an anonymous function will do the work but it is way too much to write. A proper syntax should hide all these gory details.

      • I didn’t believe it worked in Win32…
        I was thinking it was some sort of .NET extension like Array[,] …
        but this way of array-constructor call works even in Delphi 2006…
        The Create is red underlined, but it works.

        If you look at the code, it’s a lot of code…
        the SetLength and the n assignments are inlined,
        but it works.

  5. I liked your suggestions (both). The dynamic array assignment form is something would be implemented a long time ago. Btw, the anywhere-variable-assinment is something I’d like to see in OP for too long. After all, why can’t we do:

    var
    x: integer = 100;

    in local variables too? It’s semantically the same of:

    var
    x: integer;

    begin
    x := 100;

    Isn’t it? Who knows Delphi/OP needs a pre-compiler processor…

    About the list constuctor, imho, is such a f* great idea. It’s time for OP understand the needs of modern programmers and loose a little the fear of innovating features and mainly syntax. There are several ways of extend the language without loose its identity. The Object Pascal was a big step in this way, but, for some reason, people get stucked in the stairs watching pythons and rubies going up, fearing break the Turbo-Pascal-chains.

    Let’s keep getting such good ideas like these comming up and, doing that, our old good boy Pascal won’t die of “feature starvation”.

    Dekhil

    PS.: Sorry my bad english =)

    • Your English is quite fine – it shouldn’t stop you to give feedback 🙂
      “It’s time for OP understand the needs of modern programmers and loose a little the fear of innovating features and mainly syntax. There are several ways of extend the language without loose its identity.”
      Sure – your native language didn’t borrow new words from other languages, or build new words in order to express the new realities from today? Yes, (I think so). OTOH, did your native language lose its identity? No, I don’t think so.

  6. Ok, nice idea. But I think the problem is much deeper.

    Look at Ruby/Python: Everything is an object! So simple and so powerful [..and maybe so slow :)]. Both support easy-to-read Lambda function, Delphi’s lambda implementation seems ugly [begin..end, you know…]

    I think [everything-is-an-object] + [nice-looking-lambda] = [cool-syntax]. If Delphi support such things – any your suggestion would be possible.

    Example:

    A := (for i in [0..20] yield i * i)

    A is an array, which overrides “:=” operator, so if it gets lambda function as parameter, it iterates over it and fills itself with returned values. [let’s say (…) is simplified lambda notation]

    Another question: Is it [I’m talking about dynamic/functional programming] The Delphi Way?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s