Swift build settings in Xcode to use lower 4.0.3 version instead of 5.0.1 are not enforced

I'm creating a simple Swift framework in Xcode 10.3 and trying to use a lower Swift 4.0.3 version. I went to the Xcode build settings and switch from 5.0.1 to 4.0.3 and then build the project.

I ran otool against resulted framework to make sure it's built using correct Swift version but the libraries included are still from Swift 5.0.1 (corresponds to 1001.0.82 from the output below):

otool -l SwiftyHello.framework/SwiftyHello

cmd LC_LOAD_DYLIB
cmdsize 56
name @rpath/libswiftCore.dylib (offset 24)
time stamp 2 Wed Dec 31 16:00:02 1969
current version **1001.0.82**
compatibility version 1.0.0

My toolchain is running Swift 5.0.1:

swift -version

Apple Swift version 5.0.1 (swiftlang-1001.0.82.4 clang-1001.0.46.5)
Target: x86_64-apple-darwin18.7.0

How can I make sure that the Swift framework is built using Swift 4.0.3?

from Swift build settings in Xcode to use lower 4.0.3 version instead of 5.0.1 are not enforced